[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:53 dist-test-slave-10d0 krb5kdc[11696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:53 dist-test-slave-10d0 krb5kdc[11696](info): set up 2 sockets
Jun 19 06:57:53 dist-test-slave-10d0 krb5kdc[11696](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 06:57:54.569463 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:57:54 dist-test-slave-10d0 krb5kdc[11696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:57:54.696806 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46007
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:46007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:54.816777 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:54.817116 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:54.817204 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:54.821372 11709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:54.821477 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:54.821516 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:54.821552 11709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:54.821593 11709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:54.827210 11709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:46007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:54.828536 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:54.829839 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:54.839720 11715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:54.839781 11709 server_base.cc:1034] running on GCE node
W20260619 06:57:54.839781 11717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:54.839991 11714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:54.840664 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:54.842027 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:54.843195 11709 hybrid_clock.cc:648] HybridClock initialized: now 1781852274843172 us; error 38 us; skew 500 ppm
Jun 19 06:57:54 dist-test-slave-10d0 krb5kdc[11696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852274, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:54.849274 11709 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:57:54.850322 11709 webserver.cc:460] Webserver started at http://127.11.106.126:38885/ using document root <none> and password file <none>
I20260619 06:57:54.850980 11709 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:54.851045 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:54.851296 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:54.853760 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "683beb1b754448aa83cdbeef94e90058"
format_stamp: "Formatted at 2026-06-19 06:57:54 on dist-test-slave-10d0"
server_key: "c91787ec29253f6aca852962919295d4"
server_key_iv: "e93199f69a7f1d16e3d190da1a069d07"
server_key_version: "encryptionkey@0"
I20260619 06:57:54.854476 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "683beb1b754448aa83cdbeef94e90058"
format_stamp: "Formatted at 2026-06-19 06:57:54 on dist-test-slave-10d0"
server_key: "c91787ec29253f6aca852962919295d4"
server_key_iv: "e93199f69a7f1d16e3d190da1a069d07"
server_key_version: "encryptionkey@0"
I20260619 06:57:54.859244 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:54.862221 11724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:54.863705 11709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:57:54.863883 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "683beb1b754448aa83cdbeef94e90058"
format_stamp: "Formatted at 2026-06-19 06:57:54 on dist-test-slave-10d0"
server_key: "c91787ec29253f6aca852962919295d4"
server_key_iv: "e93199f69a7f1d16e3d190da1a069d07"
server_key_version: "encryptionkey@0"
I20260619 06:57:54.864058 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:54.883536 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:54.887167 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:54.887425 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:54.895987 11709 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:46007
I20260619 06:57:54.896003 11776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:46007 every 8 connection(s)
I20260619 06:57:54.897096 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:54.900346 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:54.904050 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 11709
I20260619 06:57:54.904182 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:57:54.905666 11689 external_mini_cluster.cc:1428] Setting key e33dadc6030f1540e0af0348bbb8bffe
I20260619 06:57:54.906893 11777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058: Bootstrap starting.
I20260619 06:57:54.910261 11777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:54.911357 11777 log.cc:824] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:54.913712 11777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058: No bootstrap required, opened a new log
I20260619 06:57:54.917856 11777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683beb1b754448aa83cdbeef94e90058" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46007 } }
I20260619 06:57:54.918071 11777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:54.918167 11777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683beb1b754448aa83cdbeef94e90058, State: Initialized, Role: FOLLOWER
I20260619 06:57:54.918638 11777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683beb1b754448aa83cdbeef94e90058" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46007 } }
I20260619 06:57:54.918758 11777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:54.918849 11777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:54.918970 11777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:54.919917 11777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683beb1b754448aa83cdbeef94e90058" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46007 } }
I20260619 06:57:54.920373 11777 leader_election.cc:302] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 683beb1b754448aa83cdbeef94e90058; no voters:
I20260619 06:57:54.920770 11777 leader_election.cc:288] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:54.920887 11782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:54.921131 11782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [term 1 LEADER]: Becoming Leader. State: Replica: 683beb1b754448aa83cdbeef94e90058, State: Running, Role: LEADER
I20260619 06:57:54.921525 11782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683beb1b754448aa83cdbeef94e90058" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46007 } }
I20260619 06:57:54.922112 11777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:54.923290 11784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 683beb1b754448aa83cdbeef94e90058. Latest consensus state: current_term: 1 leader_uuid: "683beb1b754448aa83cdbeef94e90058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683beb1b754448aa83cdbeef94e90058" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46007 } } }
I20260619 06:57:54.923411 11784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:54.923290 11783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "683beb1b754448aa83cdbeef94e90058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "683beb1b754448aa83cdbeef94e90058" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46007 } } }
I20260619 06:57:54.923866 11783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:54.924307 11791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 06:57:54 dist-test-slave-10d0 krb5kdc[11696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:57:54.928845 11791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:54.933842 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:54.909337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.11.106.126:46007)
0619 06:57:54.909792 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:54.909813 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:54.911001 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 06:57:54.914675 (+ 3674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:54.914692 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:54.915010 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:54.915364 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:54.917787 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:54.919314 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:54.920625 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:54.921342 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:54.922827 (+ 1485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:54.927146 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:54.927153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:54.927172 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:54.927208 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:54.929602 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:54.930308 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:54.930314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:54.930319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:54.930490 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:54.930938 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:54.930942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:54.930943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:54.931371 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:54.931584 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:54.932183 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 06:57:54.932472 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":139,"threads_started":1}
I20260619 06:57:54.936359 11791 catalog_manager.cc:1347] Generated new cluster ID: 0877a1cd31f94d9b818e00822f5c10fb
I20260619 06:57:54.936427 11791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:54.947664 11791 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:54.948876 11791 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:54.955516 11791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 683beb1b754448aa83cdbeef94e90058: Generated new TSK 0
I20260619 06:57:54.956430 11791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:57:54.987484 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46007
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:55.110478 11803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:55.110788 11803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:55.110836 11803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:55.114943 11803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:55.115010 11803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:55.115074 11803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:57:55.120563 11803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:55.121716 11803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:55.122897 11803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:55.133391 11811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:55.133517 11809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:55.133385 11808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:55.133626 11803 server_base.cc:1034] running on GCE node
I20260619 06:57:55.134193 11803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:55.134965 11803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:55.136161 11803 hybrid_clock.cc:648] HybridClock initialized: now 1781852275136138 us; error 33 us; skew 500 ppm
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:55.139086 11803 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:57:55.140126 11803 webserver.cc:460] Webserver started at http://127.11.106.65:45911/ using document root <none> and password file <none>
I20260619 06:57:55.140841 11803 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:55.140941 11803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:55.141201 11803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:55.143471 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "56498abfec164d57af4e24b0e68eb6aa"
format_stamp: "Formatted at 2026-06-19 06:57:55 on dist-test-slave-10d0"
server_key: "fe3fe7071e2348e383c47431a44aa3e4"
server_key_iv: "8ba6a06eacbf374eac7765be0a541f75"
server_key_version: "encryptionkey@0"
I20260619 06:57:55.144142 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "56498abfec164d57af4e24b0e68eb6aa"
format_stamp: "Formatted at 2026-06-19 06:57:55 on dist-test-slave-10d0"
server_key: "fe3fe7071e2348e383c47431a44aa3e4"
server_key_iv: "8ba6a06eacbf374eac7765be0a541f75"
server_key_version: "encryptionkey@0"
I20260619 06:57:55.148422 11803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:57:55.151342 11818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:55.152863 11803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:57:55.153023 11803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "56498abfec164d57af4e24b0e68eb6aa"
format_stamp: "Formatted at 2026-06-19 06:57:55 on dist-test-slave-10d0"
server_key: "fe3fe7071e2348e383c47431a44aa3e4"
server_key_iv: "8ba6a06eacbf374eac7765be0a541f75"
server_key_version: "encryptionkey@0"
I20260619 06:57:55.153184 11803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:55.169211 11803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:55.172778 11803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:55.173009 11803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:55.173697 11803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:55.174722 11803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:55.174799 11803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:55.174873 11803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:55.174918 11803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:55.186271 11803 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:46335
I20260619 06:57:55.186295 11931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:46335 every 8 connection(s)
I20260619 06:57:55.187290 11803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:57:55.194818 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 11803
I20260619 06:57:55.194906 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:57:55.195190 11689 external_mini_cluster.cc:1428] Setting key d415cd2d340962c9a9ee5e1b8e6089ce
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:57:55.202893 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:55.189413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:58897 (local address 127.11.106.126:46007)
0619 06:57:55.189594 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:55.189600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:55.190633 (+ 1033us) server_negotiation.cc:407] Connection header received
0619 06:57:55.192023 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:55.192026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:55.192083 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:55.192212 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:55.193592 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.194362 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.195671 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.195990 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.196115 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:55.198701 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:55.198705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:55.198707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:55.198733 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:55.200355 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.201101 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.201104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.201105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.201191 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.201610 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.201613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.201614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.201806 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:55.201879 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:55.202609 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 06:57:55.202728 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:57:55.204157 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:55.189949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46007 (local address 127.11.106.65:58897)
0619 06:57:55.190449 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:55.190496 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:55.191777 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:55.192445 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:55.192453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:55.192906 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:55.193418 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:55.193432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.194488 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.194491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:55.195524 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:55.195534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.196155 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.196160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:55.196346 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:55.196998 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:55.197014 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:55.198520 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:55.200486 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:55.200497 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:55.200598 (+ 101us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:55.200996 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:55.201291 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:55.201297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:55.201302 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:55.201507 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:55.201915 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:55.201926 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:55.202319 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 06:57:55.202577 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 06:57:55.202975 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 06:57:55.205572 11932 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46007
I20260619 06:57:55.205858 11932 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:55.206450 11932 heartbeater.cc:511] Master 127.11.106.126:46007 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 06:57:55.208330 11741 ts_manager.cc:194] Registered new tserver with Master: 56498abfec164d57af4e24b0e68eb6aa (127.11.106.65:46335)
I20260619 06:57:55.209550 11741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:58897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:57:55.224840 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46007
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:55.344997 11937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:55.345281 11937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:55.345368 11937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:55.349555 11937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:55.349611 11937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:55.349725 11937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:57:55.355078 11937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:55.356164 11937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:55.357276 11937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:57:55.367537 11937 server_base.cc:1034] running on GCE node
W20260619 06:57:55.367610 11945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:55.367621 11942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:55.367795 11943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:55.368353 11937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:55.369035 11937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:55.370221 11937 hybrid_clock.cc:648] HybridClock initialized: now 1781852275370211 us; error 26 us; skew 500 ppm
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:55.373185 11937 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:57:55.374029 11937 webserver.cc:460] Webserver started at http://127.11.106.66:36321/ using document root <none> and password file <none>
I20260619 06:57:55.374536 11937 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:55.374594 11937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:55.374783 11937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:55.376820 11937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "a6da3df85ea24a9ca7fa54ae546fe6c8"
format_stamp: "Formatted at 2026-06-19 06:57:55 on dist-test-slave-10d0"
server_key: "d8d36786cb10a94a96a9f9ee8f990728"
server_key_iv: "87c8f6d431396ec8a36ef48fb78e5b6f"
server_key_version: "encryptionkey@0"
I20260619 06:57:55.377344 11937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "a6da3df85ea24a9ca7fa54ae546fe6c8"
format_stamp: "Formatted at 2026-06-19 06:57:55 on dist-test-slave-10d0"
server_key: "d8d36786cb10a94a96a9f9ee8f990728"
server_key_iv: "87c8f6d431396ec8a36ef48fb78e5b6f"
server_key_version: "encryptionkey@0"
I20260619 06:57:55.381165 11937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:57:55.383517 11952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:55.384832 11937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:57:55.384927 11937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "a6da3df85ea24a9ca7fa54ae546fe6c8"
format_stamp: "Formatted at 2026-06-19 06:57:55 on dist-test-slave-10d0"
server_key: "d8d36786cb10a94a96a9f9ee8f990728"
server_key_iv: "87c8f6d431396ec8a36ef48fb78e5b6f"
server_key_version: "encryptionkey@0"
I20260619 06:57:55.385061 11937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:55.398144 11937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:55.401579 11937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:55.401782 11937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:55.402390 11937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:55.403228 11937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:55.403276 11937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:55.403342 11937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:55.403375 11937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:55.413228 11937 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:39015
I20260619 06:57:55.413259 12065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:39015 every 8 connection(s)
I20260619 06:57:55.414215 11937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:57:55.421759 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 11937
I20260619 06:57:55.421875 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:57:55.422186 11689 external_mini_cluster.cc:1428] Setting key f2f94dace13a8360bc83d3c4a5b32d02
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:57:55.430154 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:55.416105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:48413 (local address 127.11.106.126:46007)
0619 06:57:55.416256 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:55.416260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:55.417263 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 06:57:55.418544 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:55.418547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:55.418594 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:55.418670 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:55.419838 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.420357 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.421716 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.422115 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.422230 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:55.425878 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:55.425884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:55.425887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:55.425922 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:55.427734 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.428345 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.428351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.428356 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.428452 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.428891 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.428899 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.428902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.429156 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:55.429266 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:55.429770 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 06:57:55.429919 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:57:55.430807 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:55.416625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46007 (local address 127.11.106.66:48413)
0619 06:57:55.417106 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:55.417151 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:55.418320 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:55.418847 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:55.418854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:55.419213 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:55.419698 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:55.419711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.420460 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.420462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:55.421578 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:55.421588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.422311 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.422317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:55.422525 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:55.423398 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:55.423418 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:55.425680 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:55.427856 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:55.427860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:55.427873 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:55.428218 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:55.428580 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:55.428585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:55.428587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:55.428761 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:55.429267 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:55.429272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:55.429503 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 06:57:55.429652 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:57:55.429927 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":105,"threads_started":1}
I20260619 06:57:55.432001 12066 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46007
I20260619 06:57:55.432265 12066 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:55.432914 12066 heartbeater.cc:511] Master 127.11.106.126:46007 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 06:57:55.434089 11741 ts_manager.cc:194] Registered new tserver with Master: a6da3df85ea24a9ca7fa54ae546fe6c8 (127.11.106.66:39015)
I20260619 06:57:55.434700 11741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:48413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:57:55.451993 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46007
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:55.571420 12071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:55.571694 12071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:55.571738 12071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:55.575878 12071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:55.575941 12071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:55.576002 12071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:57:55.581374 12071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46007
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:55.582383 12071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:55.583365 12071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:55.593283 12077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:55.593374 12076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:55.593335 12071 server_base.cc:1034] running on GCE node
W20260619 06:57:55.593283 12079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:55.594036 12071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:55.594600 12071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:55.595752 12071 hybrid_clock.cc:648] HybridClock initialized: now 1781852275595738 us; error 28 us; skew 500 ppm
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:55.598451 12071 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:57:55.599291 12071 webserver.cc:460] Webserver started at http://127.11.106.67:39673/ using document root <none> and password file <none>
I20260619 06:57:55.599860 12071 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:55.599921 12071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:55.600121 12071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:55.602600 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "49ea537a9a30480c884d4fb31fe293cf"
format_stamp: "Formatted at 2026-06-19 06:57:55 on dist-test-slave-10d0"
server_key: "648482102ea3bdf365e988f556b39cfb"
server_key_iv: "156226326f73bfc1deac84a453acac3d"
server_key_version: "encryptionkey@0"
I20260619 06:57:55.603260 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "49ea537a9a30480c884d4fb31fe293cf"
format_stamp: "Formatted at 2026-06-19 06:57:55 on dist-test-slave-10d0"
server_key: "648482102ea3bdf365e988f556b39cfb"
server_key_iv: "156226326f73bfc1deac84a453acac3d"
server_key_version: "encryptionkey@0"
I20260619 06:57:55.608459 12071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 06:57:55.611342 12086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:55.612644 12071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 06:57:55.612761 12071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "49ea537a9a30480c884d4fb31fe293cf"
format_stamp: "Formatted at 2026-06-19 06:57:55 on dist-test-slave-10d0"
server_key: "648482102ea3bdf365e988f556b39cfb"
server_key_iv: "156226326f73bfc1deac84a453acac3d"
server_key_version: "encryptionkey@0"
I20260619 06:57:55.612888 12071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:55.629042 12071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:55.631933 12071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:55.632144 12071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:55.632781 12071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:55.633627 12071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:55.633674 12071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:55.633744 12071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:55.633786 12071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:55.643890 12071 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:40585
I20260619 06:57:55.643913 12199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:40585 every 8 connection(s)
I20260619 06:57:55.644906 12071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:57:55.648252 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 12071
I20260619 06:57:55.648394 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:57:55.648741 11689 external_mini_cluster.cc:1428] Setting key 4eaea83a048997d94fc3a2df7c99b6d1
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:57:55.659593 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:55.646854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:46903 (local address 127.11.106.126:46007)
0619 06:57:55.647001 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:55.647006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:55.647844 (+ 838us) server_negotiation.cc:407] Connection header received
0619 06:57:55.649428 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:55.649431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:55.649477 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:55.649577 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:55.650944 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.651659 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.653006 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.653239 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.653335 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:55.655873 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:55.655876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:55.655878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:55.655913 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:55.657623 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.658158 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.658161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.658162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.658240 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.658561 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.658563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.658564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.658739 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:55.658846 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:55.659351 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 06:57:55.659467 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:57:55.660449 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:55.647234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46007 (local address 127.11.106.67:46903)
0619 06:57:55.647686 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:55.647729 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:55.649198 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:55.649790 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:55.649800 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:55.650199 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:55.650811 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:55.650825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.651816 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.651819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:55.652875 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:55.652886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.653383 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.653385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:55.653539 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:55.654162 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:55.654179 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:55.655689 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:55.657750 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:55.657757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:55.657782 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:55.658060 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:55.658324 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:55.658326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:55.658327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:55.658479 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:55.658858 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:55.658863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:55.659112 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 06:57:55.659289 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 06:57:55.659556 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260619 06:57:55.661697 12200 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46007
I20260619 06:57:55.661960 12200 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:55.662639 12200 heartbeater.cc:511] Master 127.11.106.126:46007 requested a full tablet report, sending...
I20260619 06:57:55.663725 11741 ts_manager.cc:194] Registered new tserver with Master: 49ea537a9a30480c884d4fb31fe293cf (127.11.106.67:40585)
I20260619 06:57:55.664285 11741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:46903
I20260619 06:57:55.675262 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:57:55.701967 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:55.692631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.11.106.65:46335)
0619 06:57:55.692980 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:55.692994 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:55.693020 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:57:55.693096 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:55.693101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:55.693291 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:55.693418 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:55.694050 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.694790 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.695603 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.695970 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.696049 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:55.698397 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:55.698406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:55.698420 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:55.698443 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:55.700079 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.700581 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.700591 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.700592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.700688 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.700972 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.700975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.700977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.701220 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:55.701429 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:55.701614 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 06:57:55.701771 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":151,"threads_started":1}
W20260619 06:57:55.702987 11846 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57358
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[11696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:57:55.728924 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:55.720281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.11.106.65:46335)
0619 06:57:55.720438 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:55.720443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:55.720639 (+ 196us) server_negotiation.cc:407] Connection header received
0619 06:57:55.720759 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:55.720763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:55.720810 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:55.720884 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:55.721529 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.722137 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.723054 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:55.723337 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:55.723422 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:55.726037 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:55.726040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:55.726042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:55.726065 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:55.727357 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.727716 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.727719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.727721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.727794 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:55.728014 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:55.728016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:55.728017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:55.728254 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:55.728354 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:55.728593 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 06:57:55.728704 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:57:55.729991 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 11803
I20260619 06:57:55.738026 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 11937
I20260619 06:57:55.744115 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 12071
I20260619 06:57:55.749298 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 11709
2026-06-19T06:57:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2256 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[12214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[12214](info): set up 2 sockets
Jun 19 06:57:55 dist-test-slave-10d0 krb5kdc[12214](info): commencing operation
krb5kdc: starting...
W20260619 06:57:56.800778 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:57:56 dist-test-slave-10d0 krb5kdc[12214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:57:56.923835 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45809
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:57.041211 12227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:57.041574 12227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:57.041675 12227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:57.045869 12227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:57.045982 12227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:57.046026 12227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:57.046095 12227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:57.046147 12227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:57.051782 12227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:45809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:57.053139 12227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:57.054279 12227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:57.063485 12232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:57.063500 12233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:57.063482 12235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:57.063905 12227 server_base.cc:1034] running on GCE node
I20260619 06:57:57.064492 12227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:57.065656 12227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:57.066859 12227 hybrid_clock.cc:648] HybridClock initialized: now 1781852277066840 us; error 31 us; skew 500 ppm
Jun 19 06:57:57 dist-test-slave-10d0 krb5kdc[12214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:57.069586 12227 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:57:57.070529 12227 webserver.cc:460] Webserver started at http://127.11.106.126:38009/ using document root <none> and password file <none>
I20260619 06:57:57.071108 12227 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:57.071201 12227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:57.071439 12227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:57.073580 12227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "982ca4df04084cbbb6b681e75ba8d967"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "72faf20abf718a51f1477ee7949bdbf1"
server_key_iv: "e8982931a90a2009eb5afb8fa43c308a"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.074137 12227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "982ca4df04084cbbb6b681e75ba8d967"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "72faf20abf718a51f1477ee7949bdbf1"
server_key_iv: "e8982931a90a2009eb5afb8fa43c308a"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.077965 12227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:57:57.080281 12242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.081544 12227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:57:57.081683 12227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "982ca4df04084cbbb6b681e75ba8d967"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "72faf20abf718a51f1477ee7949bdbf1"
server_key_iv: "e8982931a90a2009eb5afb8fa43c308a"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.081825 12227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:57.098096 12227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:57.101254 12227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:57.101459 12227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:57.108515 12227 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:45809
I20260619 06:57:57.108632 12294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:45809 every 8 connection(s)
I20260619 06:57:57.109561 12227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:57.110565 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 12227
I20260619 06:57:57.110679 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:57:57.110948 11689 external_mini_cluster.cc:1428] Setting key 58d0d820955ba07bdb6d54cdbeb1f1db
I20260619 06:57:57.112970 12295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:57.119637 12295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967: Bootstrap starting.
Jun 19 06:57:57 dist-test-slave-10d0 krb5kdc[12214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:57:57.122488 12295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:57.123427 12295 log.cc:824] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:57.125895 12295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967: No bootstrap required, opened a new log
I20260619 06:57:57.126205 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.112368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.11.106.126:45809)
0619 06:57:57.112875 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:57.112892 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:57.112936 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:57:57.113621 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:57.113637 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:57.113977 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:57.114271 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:57.115124 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.116125 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.117255 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.117708 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.117833 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.120797 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:57.120806 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:57.120825 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:57.120863 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:57.122843 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.123400 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.123409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.123415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.123595 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.123975 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.123978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.123980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.124449 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:57.124676 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:57.124947 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 06:57:57.125244 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":199,"threads_started":1}
I20260619 06:57:57.129957 12295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982ca4df04084cbbb6b681e75ba8d967" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } }
I20260619 06:57:57.130187 12295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:57.130223 12295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982ca4df04084cbbb6b681e75ba8d967, State: Initialized, Role: FOLLOWER
I20260619 06:57:57.130676 12295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982ca4df04084cbbb6b681e75ba8d967" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } }
I20260619 06:57:57.130807 12295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:57.130851 12295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:57.130923 12295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:57.131837 12295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982ca4df04084cbbb6b681e75ba8d967" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } }
I20260619 06:57:57.132143 12295 leader_election.cc:302] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 982ca4df04084cbbb6b681e75ba8d967; no voters:
I20260619 06:57:57.132417 12295 leader_election.cc:288] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:57.132601 12300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:57.133185 12300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [term 1 LEADER]: Becoming Leader. State: Replica: 982ca4df04084cbbb6b681e75ba8d967, State: Running, Role: LEADER
I20260619 06:57:57.133520 12300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982ca4df04084cbbb6b681e75ba8d967" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } }
I20260619 06:57:57.133870 12295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:57.135406 12302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 982ca4df04084cbbb6b681e75ba8d967. Latest consensus state: current_term: 1 leader_uuid: "982ca4df04084cbbb6b681e75ba8d967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982ca4df04084cbbb6b681e75ba8d967" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } } }
I20260619 06:57:57.135553 12302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:57.135411 12301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "982ca4df04084cbbb6b681e75ba8d967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "982ca4df04084cbbb6b681e75ba8d967" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } } }
I20260619 06:57:57.135825 12301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967 [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:57.135972 12312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:57.139345 12312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:57.144991 12312 catalog_manager.cc:1347] Generated new cluster ID: 00ceea5ca93b455287941efe728f83c9
I20260619 06:57:57.145054 12312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:57.159312 12312 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:57.160327 12312 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:57.165489 12312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 982ca4df04084cbbb6b681e75ba8d967: Generated new TSK 0
I20260619 06:57:57.166230 12312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:57:57.200093 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:57.323318 12321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:57.323580 12321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:57.323623 12321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:57.327613 12321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:57.327679 12321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:57.327739 12321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:57:57.332875 12321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:57.333871 12321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:57.334947 12321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:57:57.344919 12321 server_base.cc:1034] running on GCE node
W20260619 06:57:57.344830 12327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:57.344834 12326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:57.344830 12329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:57.345541 12321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:57.346210 12321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:57.347381 12321 hybrid_clock.cc:648] HybridClock initialized: now 1781852277347362 us; error 31 us; skew 500 ppm
Jun 19 06:57:57 dist-test-slave-10d0 krb5kdc[12214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:57.350226 12321 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:57:57.351068 12321 webserver.cc:460] Webserver started at http://127.11.106.65:44799/ using document root <none> and password file <none>
I20260619 06:57:57.351656 12321 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:57.351745 12321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:57.351980 12321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:57.354136 12321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "8ffbc9e4063b44329ef0fe8f397ac039"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "6ad92b5e0c186b6bf8e9bdecb5985864"
server_key_iv: "22fdf64bbb1c85c2a0038cf71aee5f77"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.354738 12321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "8ffbc9e4063b44329ef0fe8f397ac039"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "6ad92b5e0c186b6bf8e9bdecb5985864"
server_key_iv: "22fdf64bbb1c85c2a0038cf71aee5f77"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.358759 12321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:57:57.361536 12336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.362862 12321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:57:57.363022 12321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "8ffbc9e4063b44329ef0fe8f397ac039"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "6ad92b5e0c186b6bf8e9bdecb5985864"
server_key_iv: "22fdf64bbb1c85c2a0038cf71aee5f77"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.363145 12321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:57.379108 12321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:57.382395 12321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:57.382597 12321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:57.383184 12321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:57.383998 12321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:57.384069 12321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.384140 12321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:57.384193 12321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.393589 12321 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:36603
I20260619 06:57:57.393671 12449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:36603 every 8 connection(s)
I20260619 06:57:57.394575 12321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:57:57.397471 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 12321
I20260619 06:57:57.397580 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:57:57.397854 11689 external_mini_cluster.cc:1428] Setting key 40f3017426324141d2c397c69fb2724e
Jun 19 06:57:57 dist-test-slave-10d0 krb5kdc[12214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 06:57:57.410549 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.396517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:56721 (local address 127.11.106.126:45809)
0619 06:57:57.396733 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:57.396739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:57.397622 (+ 883us) server_negotiation.cc:407] Connection header received
0619 06:57:57.399028 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:57.399031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:57.399086 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:57.399190 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:57.400370 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.401276 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.402864 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.403340 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.403466 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.406448 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:57.406451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:57.406453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:57.406480 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:57.408085 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.408767 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.408771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.408773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.408859 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.409320 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.409323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.409325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.409522 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:57.409633 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:57.410293 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 06:57:57.410421 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 06:57:57.411826 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.396966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45809 (local address 127.11.106.65:56721)
0619 06:57:57.397458 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:57.397501 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:57.398815 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:57.399378 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:57.399386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:57.399752 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:57.400220 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.400233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.401429 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.401432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.402720 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.402730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.403536 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.403540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.403756 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.404420 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:57.404437 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:57.406238 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:57.408240 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:57.408247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:57.408267 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:57.408657 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:57.409000 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:57.409003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:57.409005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:57.409225 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:57.409634 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:57.409641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:57.409980 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 06:57:57.410213 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 06:57:57.410624 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":109,"threads_started":1}
I20260619 06:57:57.413236 12450 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45809
I20260619 06:57:57.413511 12450 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:57.414129 12450 heartbeater.cc:511] Master 127.11.106.126:45809 requested a full tablet report, sending...
I20260619 06:57:57.416016 12259 ts_manager.cc:194] Registered new tserver with Master: 8ffbc9e4063b44329ef0fe8f397ac039 (127.11.106.65:36603)
I20260619 06:57:57.417291 12259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:56721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:57:57.428066 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:57.541296 12455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:57.541589 12455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:57.541666 12455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:57.545688 12455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:57.545780 12455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:57.545869 12455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:57:57.551149 12455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:57.552578 12455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:57.553598 12455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:57.563516 12461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:57.563517 12463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:57.563519 12460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:57.563575 12455 server_base.cc:1034] running on GCE node
I20260619 06:57:57.564249 12455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:57.564953 12455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:57.566141 12455 hybrid_clock.cc:648] HybridClock initialized: now 1781852277566116 us; error 37 us; skew 500 ppm
Jun 19 06:57:57 dist-test-slave-10d0 krb5kdc[12214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:57.569186 12455 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:57:57.570065 12455 webserver.cc:460] Webserver started at http://127.11.106.66:43603/ using document root <none> and password file <none>
I20260619 06:57:57.570650 12455 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:57.570746 12455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:57.570981 12455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:57.573269 12455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "251c1c32661445019271e45e2ab4c2ba"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "8bb55c6850ed33a17b60336c735b17fd"
server_key_iv: "f7f5bf9f09a5ff46966a6cfda4ce2ec6"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.573858 12455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "251c1c32661445019271e45e2ab4c2ba"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "8bb55c6850ed33a17b60336c735b17fd"
server_key_iv: "f7f5bf9f09a5ff46966a6cfda4ce2ec6"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.577983 12455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:57:57.580487 12470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.581750 12455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:57:57.581902 12455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "251c1c32661445019271e45e2ab4c2ba"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "8bb55c6850ed33a17b60336c735b17fd"
server_key_iv: "f7f5bf9f09a5ff46966a6cfda4ce2ec6"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.582005 12455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:57.606115 12455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:57.609439 12455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:57.609654 12455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:57.610301 12455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:57.611198 12455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:57.611270 12455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.611338 12455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:57.611392 12455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.620242 12455 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:36575
I20260619 06:57:57.620265 12583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:36575 every 8 connection(s)
I20260619 06:57:57.621202 12455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:57:57.625154 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 12455
I20260619 06:57:57.625274 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:57:57.625566 11689 external_mini_cluster.cc:1428] Setting key a19f76427ac7198b514a194659713dd7
Jun 19 06:57:57 dist-test-slave-10d0 krb5kdc[12214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 06:57:57.637100 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.623064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:38963 (local address 127.11.106.126:45809)
0619 06:57:57.623212 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:57.623225 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:57.624075 (+ 850us) server_negotiation.cc:407] Connection header received
0619 06:57:57.625653 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:57.625658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:57.625723 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:57.625791 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:57.627264 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.627778 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.629410 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.629640 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.629772 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.632916 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:57.632919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:57.632921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:57.632945 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:57.634421 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.635224 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.635227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.635228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.635304 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.635721 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.635723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.635724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.636051 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:57.636131 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:57.636865 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 06:57:57.636976 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:57:57.638128 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.623468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45809 (local address 127.11.106.66:38963)
0619 06:57:57.623918 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:57.623961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:57.625370 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:57.626019 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:57.626030 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:57.626478 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:57.627114 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.627137 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.627919 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.627925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.629275 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.629285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.629819 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.629823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.630056 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.630711 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:57.630740 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:57.632712 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:57.634546 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:57.634556 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:57.634581 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:57.635104 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:57.635402 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:57.635405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:57.635408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:57.635622 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:57.636154 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:57.636160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:57.636522 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 06:57:57.636779 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 06:57:57.637067 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 06:57:57.639490 12584 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45809
I20260619 06:57:57.639806 12584 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:57.640359 12584 heartbeater.cc:511] Master 127.11.106.126:45809 requested a full tablet report, sending...
I20260619 06:57:57.641571 12259 ts_manager.cc:194] Registered new tserver with Master: 251c1c32661445019271e45e2ab4c2ba (127.11.106.66:36575)
I20260619 06:57:57.642462 12259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:38963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:57:57.657045 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:57.771503 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:57.771770 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:57.771816 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:57.775878 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:57.775940 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:57.776000 12589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:57:57.781338 12589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:57.782308 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:57.783264 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:57.792732 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:57.792781 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:57.792747 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:57.793092 12589 server_base.cc:1034] running on GCE node
I20260619 06:57:57.793500 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:57.794137 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:57.795500 12589 hybrid_clock.cc:648] HybridClock initialized: now 1781852277795472 us; error 53 us; skew 500 ppm
Jun 19 06:57:57 dist-test-slave-10d0 krb5kdc[12214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:57.798426 12589 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:57:57.799420 12589 webserver.cc:460] Webserver started at http://127.11.106.67:41765/ using document root <none> and password file <none>
I20260619 06:57:57.799983 12589 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:57.800071 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:57.800297 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:57.802378 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "cabb0d196bb840ca9e70e5856ec2dfb1"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "57bcb016fccb74cb79b5427e2d948f74"
server_key_iv: "e9f94f5fc33f648e28120dad9d7a9134"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.802953 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "cabb0d196bb840ca9e70e5856ec2dfb1"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "57bcb016fccb74cb79b5427e2d948f74"
server_key_iv: "e9f94f5fc33f648e28120dad9d7a9134"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.806602 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:57:57.808913 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.810042 12589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:57:57.810178 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "cabb0d196bb840ca9e70e5856ec2dfb1"
format_stamp: "Formatted at 2026-06-19 06:57:57 on dist-test-slave-10d0"
server_key: "57bcb016fccb74cb79b5427e2d948f74"
server_key_iv: "e9f94f5fc33f648e28120dad9d7a9134"
server_key_version: "encryptionkey@0"
I20260619 06:57:57.810297 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:57.829344 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:57.832264 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:57.832475 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:57.833117 12589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:57:57.834107 12589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:57.834185 12589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.834266 12589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:57.834308 12589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:57.843349 12589 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:39251
I20260619 06:57:57.843372 12717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:39251 every 8 connection(s)
I20260619 06:57:57.844256 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
Jun 19 06:57:57 dist-test-slave-10d0 krb5kdc[12214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852277, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:57:57.854169 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 12589
I20260619 06:57:57.854326 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:57:57.854655 11689 external_mini_cluster.cc:1428] Setting key 7d969a3cd6e15ee1539f685407bea55e
I20260619 06:57:57.858534 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.846158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:33205 (local address 127.11.106.126:45809)
0619 06:57:57.846301 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:57.846306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:57.847151 (+ 845us) server_negotiation.cc:407] Connection header received
0619 06:57:57.848427 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:57.848431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:57.848489 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:57.848654 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:57.849794 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.850330 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.851458 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.851688 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.851799 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.854311 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:57.854314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:57.854316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:57.854340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:57.856137 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.856703 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.856706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.856708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.856800 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.857343 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.857347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.857348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.857568 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:57.857717 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:57.858290 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 06:57:57.858415 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:57:57.859424 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.846564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45809 (local address 127.11.106.67:33205)
0619 06:57:57.847001 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:57.847045 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:57.848216 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:57.848831 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:57.848838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:57.849186 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:57.849650 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.849663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.850456 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.850464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.851339 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.851348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.851846 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.851851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.852014 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.852634 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:57.852651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:57.854137 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:57.856269 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:57.856274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:57.856287 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:57.856611 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:57.856925 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:57.856928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:57.856929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:57.857218 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:57.857754 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:57.857760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:57.858000 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 06:57:57.858173 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 06:57:57.858466 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260619 06:57:57.860584 12718 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45809
I20260619 06:57:57.860841 12718 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:57.861413 12718 heartbeater.cc:511] Master 127.11.106.126:45809 requested a full tablet report, sending...
I20260619 06:57:57.862473 12259 ts_manager.cc:194] Registered new tserver with Master: cabb0d196bb840ca9e70e5856ec2dfb1 (127.11.106.67:39251)
I20260619 06:57:57.863003 12259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:33205
I20260619 06:57:57.868799 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:57:57.879891 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.872439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49304 (local address 127.11.106.126:45809)
0619 06:57:57.872618 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:57.872623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:57.872792 (+ 169us) server_negotiation.cc:407] Connection header received
0619 06:57:57.872948 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:57.872951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:57.873002 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:57.873085 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:57.873823 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.874346 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.875183 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.875401 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.875510 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.876955 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:57.876959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:57.876962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:57.876992 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:57.878211 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.878638 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.878643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.878645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.878728 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:57.879010 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:57.879013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:57.879015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:57.879235 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:57.879375 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:57.879613 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 06:57:57.879727 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:57:57.884831 12259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:57:57.888041 12259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:57:57.903258 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.897842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.11.106.66:36575)
0619 06:57:57.898177 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:57.898192 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:57.898410 (+ 218us) server_negotiation.cc:407] Connection header received
0619 06:57:57.898615 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:57.898620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:57.898763 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:57.898894 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:57.899397 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.900517 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.901975 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.902539 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.902679 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.902912 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:57.903002 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 06:57:57.903102 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
I20260619 06:57:57.903414 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.897208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:36603 (local address 127.0.0.1:34228)
0619 06:57:57.897658 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:57.897717 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:57.897849 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:57.898971 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:57.898977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:57.899001 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:57.899136 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.899142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.900670 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.900673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.901892 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.901903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.903002 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.903006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.903090 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.903112 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:57:57.903163 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 06:57:57.903247 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":36,"thread_start_us":93,"threads_started":1}
I20260619 06:57:57.903249 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.897369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:36575 (local address 127.0.0.1:57250)
0619 06:57:57.898301 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:57.898315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:57.898484 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:57.899119 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:57.899121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:57.899137 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:57.899265 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.899272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.900721 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.900729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.901856 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.901866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.902668 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.902672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.902807 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.902849 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:57:57.902970 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 06:57:57.903094 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":50,"thread_start_us":114,"threads_started":1}
I20260619 06:57:57.903788 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.897341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.11.106.65:36603)
0619 06:57:57.898406 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:57.898420 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:57.898440 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:57:57.898520 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:57.898529 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:57.898758 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:57.898936 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:57.899276 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.900517 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.902036 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.902863 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.903338 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.903494 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:57.903541 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 06:57:57.903627 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":94,"threads_started":1}
I20260619 06:57:57.904233 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.898138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.11.106.67:39251)
0619 06:57:57.898559 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:57.898571 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:57.898926 (+ 355us) server_negotiation.cc:407] Connection header received
0619 06:57:57.899002 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:57.899007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:57.899186 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:57.899336 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:57.899498 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.900218 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.902036 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.902870 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.902979 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.903113 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:57.903955 (+ 842us) server_negotiation.cc:299] Negotiation successful
0619 06:57:57.904062 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":100,"threads_started":1}
I20260619 06:57:57.906540 12518 tablet_service.cc:1467] Processing CreateTablet for tablet 29a42a5b8daf4669a4a5eaadab4e5927 (DEFAULT_TABLE table=test-table [id=31788e6d0c144910a7ae3b336cb49e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:57.907356 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:57.897978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:39251 (local address 127.0.0.1:33724)
0619 06:57:57.898697 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:57.898711 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:57.898814 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:57.899307 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:57.899310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:57.899327 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:57.899411 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.899418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.900354 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.900361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.901892 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:57.901903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:57.903171 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:57.903174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:57.903293 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:57.903313 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:57:57.903764 (+ 451us) client_negotiation.cc:240] Negotiation successful
0619 06:57:57.907205 (+ 3441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":72,"thread_start_us":95,"threads_started":1}
I20260619 06:57:57.907675 12518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a42a5b8daf4669a4a5eaadab4e5927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:57.908452 12384 tablet_service.cc:1467] Processing CreateTablet for tablet 29a42a5b8daf4669a4a5eaadab4e5927 (DEFAULT_TABLE table=test-table [id=31788e6d0c144910a7ae3b336cb49e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:57.909543 12384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a42a5b8daf4669a4a5eaadab4e5927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:57.910281 12652 tablet_service.cc:1467] Processing CreateTablet for tablet 29a42a5b8daf4669a4a5eaadab4e5927 (DEFAULT_TABLE table=test-table [id=31788e6d0c144910a7ae3b336cb49e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:57:57.911327 12652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a42a5b8daf4669a4a5eaadab4e5927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:57.914710 12735 tablet_bootstrap.cc:492] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba: Bootstrap starting.
I20260619 06:57:57.916885 12736 tablet_bootstrap.cc:492] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039: Bootstrap starting.
I20260619 06:57:57.917443 12735 tablet_bootstrap.cc:654] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:57.918162 12737 tablet_bootstrap.cc:492] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1: Bootstrap starting.
I20260619 06:57:57.918365 12735 log.cc:824] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:57.919119 12736 tablet_bootstrap.cc:654] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:57.919940 12736 log.cc:824] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:57.920418 12737 tablet_bootstrap.cc:654] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:57.921124 12735 tablet_bootstrap.cc:492] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba: No bootstrap required, opened a new log
I20260619 06:57:57.921326 12735 ts_tablet_manager.cc:1397] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:57:57.921648 12737 log.cc:824] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:57.921850 12736 tablet_bootstrap.cc:492] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039: No bootstrap required, opened a new log
I20260619 06:57:57.922062 12736 ts_tablet_manager.cc:1397] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:57:57.923557 12737 tablet_bootstrap.cc:492] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1: No bootstrap required, opened a new log
I20260619 06:57:57.923732 12737 ts_tablet_manager.cc:1397] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 06:57:57.925321 12735 raft_consensus.cc:348] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } }
I20260619 06:57:57.925449 12736 raft_consensus.cc:348] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } }
I20260619 06:57:57.925637 12735 raft_consensus.cc:374] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:57.925688 12735 raft_consensus.cc:729] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251c1c32661445019271e45e2ab4c2ba, State: Initialized, Role: FOLLOWER
I20260619 06:57:57.925695 12736 raft_consensus.cc:374] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:57.925774 12736 raft_consensus.cc:729] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffbc9e4063b44329ef0fe8f397ac039, State: Initialized, Role: FOLLOWER
I20260619 06:57:57.926177 12735 consensus_queue.cc:260] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } }
I20260619 06:57:57.926230 12736 consensus_queue.cc:260] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } }
I20260619 06:57:57.927109 12584 heartbeater.cc:503] Master 127.11.106.126:45809 was elected leader, sending a full tablet report...
I20260619 06:57:57.927218 12736 ts_tablet_manager.cc:1428] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:57:57.927474 12450 heartbeater.cc:503] Master 127.11.106.126:45809 was elected leader, sending a full tablet report...
I20260619 06:57:57.927610 12737 raft_consensus.cc:348] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } }
I20260619 06:57:57.927903 12737 raft_consensus.cc:374] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:57.927965 12737 raft_consensus.cc:729] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabb0d196bb840ca9e70e5856ec2dfb1, State: Initialized, Role: FOLLOWER
I20260619 06:57:57.928593 12735 ts_tablet_manager.cc:1428] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 06:57:57.928511 12737 consensus_queue.cc:260] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } }
I20260619 06:57:57.929123 12718 heartbeater.cc:503] Master 127.11.106.126:45809 was elected leader, sending a full tablet report...
I20260619 06:57:57.929330 12737 ts_tablet_manager.cc:1428] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260619 06:57:58.088972 12743 raft_consensus.cc:482] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:57:58.089200 12743 raft_consensus.cc:504] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } }
I20260619 06:57:58.090394 12743 leader_election.cc:288] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ffbc9e4063b44329ef0fe8f397ac039 (127.11.106.65:36603), 251c1c32661445019271e45e2ab4c2ba (127.11.106.66:36575)
I20260619 06:57:58.094856 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:58.090901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:45007 (local address 127.11.106.66:36575)
0619 06:57:58.091036 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:58.091041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:58.091322 (+ 281us) server_negotiation.cc:407] Connection header received
0619 06:57:58.091446 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:58.091449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:58.091492 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:58.091561 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:58.092020 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.092844 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.093892 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.094284 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.094410 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:58.094468 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:58.094653 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 06:57:58.094731 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:57:58.094857 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:58.090686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:36603 (local address 127.11.106.67:45817)
0619 06:57:58.090862 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:58.090879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:58.090983 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:58.091308 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:58.091311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:58.091342 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:58.091605 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:58.091617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.092853 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.092856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:58.093738 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:58.093747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.094426 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.094429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:58.094550 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:58.094566 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 06:57:58.094662 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:57:58.094729 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":29,"spinlock_wait_cycles":1792}
I20260619 06:57:58.094882 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:58.090796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:45817 (local address 127.11.106.65:36603)
0619 06:57:58.090930 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:58.090934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:58.090950 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:57:58.091136 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:58.091140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:58.091198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:58.091290 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:57:58.091807 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.092730 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.093865 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.094291 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.094426 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:58.094480 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:58.094690 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 06:57:58.094778 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:57:58.094867 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:58.090686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:36575 (local address 127.11.106.67:45007)
0619 06:57:58.091198 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:58.091208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:58.091342 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:58.091632 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:58.091635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:58.091656 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:57:58.091827 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:58.091834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.092943 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.092945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:58.093745 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:58.093754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.094413 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.094416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:58.094515 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:58.094543 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:57:58.094604 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 06:57:58.094688 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":151,"spinlock_wait_cycles":12416,"thread_start_us":144,"threads_started":1}
W20260619 06:57:58.095249 12719 tablet.cc:2369] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:57:58.095691 12538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a42a5b8daf4669a4a5eaadab4e5927" candidate_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251c1c32661445019271e45e2ab4c2ba" is_pre_election: true
I20260619 06:57:58.096029 12538 raft_consensus.cc:2454] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cabb0d196bb840ca9e70e5856ec2dfb1 in term 0.
I20260619 06:57:58.096163 12404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a42a5b8daf4669a4a5eaadab4e5927" candidate_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" is_pre_election: true
I20260619 06:57:58.096460 12404 raft_consensus.cc:2454] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cabb0d196bb840ca9e70e5856ec2dfb1 in term 0.
I20260619 06:57:58.096524 12608 leader_election.cc:302] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 251c1c32661445019271e45e2ab4c2ba, cabb0d196bb840ca9e70e5856ec2dfb1; no voters:
I20260619 06:57:58.096969 12743 raft_consensus.cc:2781] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:57:58.097056 12743 raft_consensus.cc:482] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:57:58.097137 12743 raft_consensus.cc:3037] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:58.098150 12743 raft_consensus.cc:504] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } }
I20260619 06:57:58.098516 12743 leader_election.cc:288] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [CANDIDATE]: Term 1 election: Requested vote from peers 8ffbc9e4063b44329ef0fe8f397ac039 (127.11.106.65:36603), 251c1c32661445019271e45e2ab4c2ba (127.11.106.66:36575)
I20260619 06:57:58.098898 12538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a42a5b8daf4669a4a5eaadab4e5927" candidate_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "251c1c32661445019271e45e2ab4c2ba"
I20260619 06:57:58.098922 12404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a42a5b8daf4669a4a5eaadab4e5927" candidate_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffbc9e4063b44329ef0fe8f397ac039"
I20260619 06:57:58.099035 12538 raft_consensus.cc:3037] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:58.099041 12404 raft_consensus.cc:3037] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:58.100073 12404 raft_consensus.cc:2454] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cabb0d196bb840ca9e70e5856ec2dfb1 in term 1.
I20260619 06:57:58.100158 12538 raft_consensus.cc:2454] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cabb0d196bb840ca9e70e5856ec2dfb1 in term 1.
I20260619 06:57:58.100384 12605 leader_election.cc:302] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ffbc9e4063b44329ef0fe8f397ac039, cabb0d196bb840ca9e70e5856ec2dfb1; no voters:
I20260619 06:57:58.100579 12743 raft_consensus.cc:2781] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:58.100804 12743 raft_consensus.cc:686] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [term 1 LEADER]: Becoming Leader. State: Replica: cabb0d196bb840ca9e70e5856ec2dfb1, State: Running, Role: LEADER
I20260619 06:57:58.101158 12743 consensus_queue.cc:237] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } }
I20260619 06:57:58.104336 12259 catalog_manager.cc:5495] T 29a42a5b8daf4669a4a5eaadab4e5927 P cabb0d196bb840ca9e70e5856ec2dfb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to cabb0d196bb840ca9e70e5856ec2dfb1 (127.11.106.67). New cstate: current_term: 1 leader_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffbc9e4063b44329ef0fe8f397ac039" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "251c1c32661445019271e45e2ab4c2ba" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 36575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cabb0d196bb840ca9e70e5856ec2dfb1" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39251 } health_report { overall_health: HEALTHY } } }
W20260619 06:57:58.122254 12585 tablet.cc:2369] T 29a42a5b8daf4669a4a5eaadab4e5927 P 251c1c32661445019271e45e2ab4c2ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:57:58 dist-test-slave-10d0 krb5kdc[12214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:57:58.141636 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:58.132592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.11.106.65:36603)
0619 06:57:58.132740 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:58.132745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:58.132928 (+ 183us) server_negotiation.cc:407] Connection header received
0619 06:57:58.133026 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:58.133028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:58.133073 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:58.133142 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:58.133788 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.134354 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.135218 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.135426 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.135503 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:58.137996 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:58.138008 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:58.138025 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:58.138048 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:58.139830 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:58.140232 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:58.140238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:58.140240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:58.140322 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:58.140650 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:58.140653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:58.140655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:58.140934 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:58.141100 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:58.141299 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 06:57:58.141427 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 06:57:58.145896 12451 tablet.cc:2369] T 29a42a5b8daf4669a4a5eaadab4e5927 P 8ffbc9e4063b44329ef0fe8f397ac039: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:57:58 dist-test-slave-10d0 krb5kdc[12214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852278, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:57:58 dist-test-slave-10d0 krb5kdc[12214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852278, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:57:58.169209 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:58.160229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.11.106.65:36603)
0619 06:57:58.160431 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:58.160435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:58.160645 (+ 210us) server_negotiation.cc:407] Connection header received
0619 06:57:58.160781 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:58.160783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:58.160827 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:58.160899 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:58.161559 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.162060 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.162840 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.163058 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.163138 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:58.165636 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:58.165639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:58.165641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:58.165708 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:58.167327 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:58.167806 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:58.167809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:58.167810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:58.167944 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:58.168236 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:58.168239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:58.168240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:58.168429 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:58.168615 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:58.168914 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 06:57:58.169042 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 06:57:58 dist-test-slave-10d0 krb5kdc[12214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:57:58 dist-test-slave-10d0 krb5kdc[12214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:57:58.194811 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:58.186541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.11.106.65:36603)
0619 06:57:58.186725 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:58.186730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:58.187093 (+ 363us) server_negotiation.cc:407] Connection header received
0619 06:57:58.187201 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:58.187203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:58.187249 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:58.187329 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:58.187933 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.188418 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.189274 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:58.189591 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:58.189671 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:58.192064 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:58.192067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:58.192069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:58.192091 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:58.193519 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:58.193874 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:58.193877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:58.193878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:58.193949 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:58.194161 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:58.194163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:58.194164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:58.194312 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:58.194405 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:58.194592 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 06:57:58.194684 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 06:57:58.196223 12364 tablet_service.cc:2908] Rejecting scan request for tablet 29a42a5b8daf4669a4a5eaadab4e5927: Uninitialized: clean time has not yet been initialized
I20260619 06:57:58.198370 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 12321
I20260619 06:57:58.204595 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 12455
I20260619 06:57:58.210477 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 12589
I20260619 06:57:58.217764 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 12227
2026-06-19T06:57:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2468 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:57:58 dist-test-slave-10d0 krb5kdc[12759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:57:58 dist-test-slave-10d0 krb5kdc[12759](info): set up 2 sockets
Jun 19 06:57:58 dist-test-slave-10d0 krb5kdc[12759](info): commencing operation
krb5kdc: starting...
W20260619 06:57:59.277097 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:57:59 dist-test-slave-10d0 krb5kdc[12759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:57:59.403630 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:39939
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:39939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:59.519227 12772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:59.519583 12772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:59.519675 12772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:59.523850 12772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:57:59.523984 12772 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:57:59.524017 12772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:59.524065 12772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:59.524101 12772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:57:59.524135 12772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:57:59.529811 12772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:39939
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:39939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:59.531011 12772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:59.532094 12772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:59.541620 12780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:59.541625 12778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:59.541620 12777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:59.541828 12772 server_base.cc:1034] running on GCE node
I20260619 06:57:59.542421 12772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:59.543699 12772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:59.544894 12772 hybrid_clock.cc:648] HybridClock initialized: now 1781852279544876 us; error 29 us; skew 500 ppm
Jun 19 06:57:59 dist-test-slave-10d0 krb5kdc[12759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852279, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:59.547758 12772 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:57:59.548763 12772 webserver.cc:460] Webserver started at http://127.11.106.126:40785/ using document root <none> and password file <none>
I20260619 06:57:59.549363 12772 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:59.549458 12772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:59.549695 12772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:59.551816 12772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "9e2f95f46ce44b8bb27405b0c6688f4d"
format_stamp: "Formatted at 2026-06-19 06:57:59 on dist-test-slave-10d0"
server_key: "160befe5d3b17facc22b4ea5d43b7cdb"
server_key_iv: "000bb63f4b45427c781f31d969761c9a"
server_key_version: "encryptionkey@0"
I20260619 06:57:59.552385 12772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "9e2f95f46ce44b8bb27405b0c6688f4d"
format_stamp: "Formatted at 2026-06-19 06:57:59 on dist-test-slave-10d0"
server_key: "160befe5d3b17facc22b4ea5d43b7cdb"
server_key_iv: "000bb63f4b45427c781f31d969761c9a"
server_key_version: "encryptionkey@0"
I20260619 06:57:59.556347 12772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 06:57:59.559046 12787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:59.560598 12772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:57:59.560745 12772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "9e2f95f46ce44b8bb27405b0c6688f4d"
format_stamp: "Formatted at 2026-06-19 06:57:59 on dist-test-slave-10d0"
server_key: "160befe5d3b17facc22b4ea5d43b7cdb"
server_key_iv: "000bb63f4b45427c781f31d969761c9a"
server_key_version: "encryptionkey@0"
I20260619 06:57:59.560878 12772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:59.585843 12772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:59.589244 12772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:59.589473 12772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:59.597939 12772 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:39939
I20260619 06:57:59.597957 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:39939 every 8 connection(s)
I20260619 06:57:59.598951 12772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:57:59.600687 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 12772
I20260619 06:57:59.600862 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:57:59.601168 11689 external_mini_cluster.cc:1428] Setting key 3c21c5cff99b5586e801648ffe1156f1
I20260619 06:57:59.602310 12850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:57:59.608171 12850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d: Bootstrap starting.
Jun 19 06:57:59 dist-test-slave-10d0 krb5kdc[12759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:57:59.610774 12850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d: Neither blocks nor log segments found. Creating new log.
I20260619 06:57:59.612087 12850 log.cc:824] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:57:59.614423 12850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d: No bootstrap required, opened a new log
I20260619 06:57:59.615648 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:59.602705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.11.106.126:39939)
0619 06:57:59.603152 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:59.603167 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:59.603204 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:57:59.603871 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:59.603888 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:59.604213 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:59.604549 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:59.605383 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.606385 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.607287 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.607825 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.608007 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:59.610742 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:59.610747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:59.610761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:59.610789 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:59.612706 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:59.613215 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:59.613220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:59.613224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:59.613335 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:59.613613 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:59.613615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:59.613616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:59.614009 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:59.614157 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:59.614451 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 06:57:59.614903 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20260619 06:57:59.617599 12850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f95f46ce44b8bb27405b0c6688f4d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39939 } }
I20260619 06:57:59.617884 12850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:57:59.617954 12850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2f95f46ce44b8bb27405b0c6688f4d, State: Initialized, Role: FOLLOWER
I20260619 06:57:59.618424 12850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f95f46ce44b8bb27405b0c6688f4d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39939 } }
I20260619 06:57:59.618568 12850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:57:59.618633 12850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:57:59.618746 12850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:57:59.619649 12850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f95f46ce44b8bb27405b0c6688f4d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39939 } }
I20260619 06:57:59.619987 12850 leader_election.cc:302] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e2f95f46ce44b8bb27405b0c6688f4d; no voters:
I20260619 06:57:59.620309 12850 leader_election.cc:288] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:57:59.620443 12855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:57:59.620712 12855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [term 1 LEADER]: Becoming Leader. State: Replica: 9e2f95f46ce44b8bb27405b0c6688f4d, State: Running, Role: LEADER
I20260619 06:57:59.621021 12855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f95f46ce44b8bb27405b0c6688f4d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39939 } }
I20260619 06:57:59.621588 12850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:57:59.622783 12857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e2f95f46ce44b8bb27405b0c6688f4d. Latest consensus state: current_term: 1 leader_uuid: "9e2f95f46ce44b8bb27405b0c6688f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f95f46ce44b8bb27405b0c6688f4d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39939 } } }
I20260619 06:57:59.622922 12857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:59.623276 12864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:57:59.623245 12856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e2f95f46ce44b8bb27405b0c6688f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f95f46ce44b8bb27405b0c6688f4d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39939 } } }
I20260619 06:57:59.624022 12856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d [sys.catalog]: This master's current role is: LEADER
I20260619 06:57:59.626483 12864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:57:59.631942 12864 catalog_manager.cc:1347] Generated new cluster ID: ecbeb763cde841308c1aef3e47a66bc2
I20260619 06:57:59.632010 12864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:57:59.652151 12864 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:57:59.653254 12864 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:57:59.658778 12864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d: Generated new TSK 0
I20260619 06:57:59.659521 12864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:57:59.693995 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39939
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:57:59.810415 12876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:57:59.810683 12876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:57:59.810768 12876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:57:59.814872 12876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:57:59.814936 12876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:57:59.814976 12876 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:57:59.815042 12876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:57:59.820431 12876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.106.126:39939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:57:59.821558 12876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:57:59.822799 12876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:57:59.833544 12884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:59.833544 12881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:57:59.833590 12882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:57:59.833788 12876 server_base.cc:1034] running on GCE node
I20260619 06:57:59.834247 12876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:57:59.835006 12876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:57:59.836201 12876 hybrid_clock.cc:648] HybridClock initialized: now 1781852279836184 us; error 32 us; skew 500 ppm
Jun 19 06:57:59 dist-test-slave-10d0 krb5kdc[12759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852279, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:57:59.839051 12876 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:57:59.840154 12876 webserver.cc:460] Webserver started at http://127.11.106.65:34943/ using document root <none> and password file <none>
I20260619 06:57:59.840816 12876 fs_manager.cc:359] Metadata directory not provided
I20260619 06:57:59.840917 12876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:57:59.841149 12876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:57:59.843240 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "f98d49dcab6040f9bf121b08c04bf554"
format_stamp: "Formatted at 2026-06-19 06:57:59 on dist-test-slave-10d0"
server_key: "d78e5e3fea47c81c1404336ad305ccc7"
server_key_iv: "25eefe472cf083396be6fd420840830b"
server_key_version: "encryptionkey@0"
I20260619 06:57:59.843825 12876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "f98d49dcab6040f9bf121b08c04bf554"
format_stamp: "Formatted at 2026-06-19 06:57:59 on dist-test-slave-10d0"
server_key: "d78e5e3fea47c81c1404336ad305ccc7"
server_key_iv: "25eefe472cf083396be6fd420840830b"
server_key_version: "encryptionkey@0"
I20260619 06:57:59.847574 12876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 06:57:59.850178 12891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:59.851502 12876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:57:59.851639 12876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "f98d49dcab6040f9bf121b08c04bf554"
format_stamp: "Formatted at 2026-06-19 06:57:59 on dist-test-slave-10d0"
server_key: "d78e5e3fea47c81c1404336ad305ccc7"
server_key_iv: "25eefe472cf083396be6fd420840830b"
server_key_version: "encryptionkey@0"
I20260619 06:57:59.851792 12876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:57:59.874698 12876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:57:59.877888 12876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:57:59.878099 12876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:57:59.879603 12876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:57:59.879683 12876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:57:59.879757 12876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:57:59.879812 12876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:57:59 dist-test-slave-10d0 krb5kdc[12759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852279, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:57:59.891395 12876 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:32789
I20260619 06:57:59.891428 13006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:32789 every 8 connection(s)
I20260619 06:57:59.892287 12876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:57:59.893966 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:59.880618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:49197 (local address 127.11.106.126:39939)
0619 06:57:59.880749 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:59.880755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:59.881966 (+ 1211us) server_negotiation.cc:407] Connection header received
0619 06:57:59.883141 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:59.883147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:59.883228 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:59.883331 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:59.884737 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.885325 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.886589 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.886884 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.887001 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:59.890053 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:59.890057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:59.890060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:59.890094 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:59.891715 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:59.892270 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:59.892273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:59.892275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:59.892393 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:59.892853 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:59.892856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:59.892858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:59.893084 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:59.893180 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:59.893726 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 06:57:59.893847 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:57:59.894721 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:59.881201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39939 (local address 127.11.106.65:49197)
0619 06:57:59.881818 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:59.881861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:59.882851 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:59.883558 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:59.883570 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:59.883964 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:59.884580 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:59.884603 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.885448 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.885451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:59.886461 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:59.886471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.887050 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.887054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:59.887282 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:59.887978 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:59.888003 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:59.889879 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:59.891833 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:59.891839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:59.891854 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:59.892155 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:59.892499 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:59.892502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:59.892504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:59.892728 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:59.893232 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:59.893238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:59.893466 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 06:57:59.893616 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 06:57:59.893870 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":132,"threads_started":1}
I20260619 06:57:59.896046 13007 heartbeater.cc:347] Connected to a master server at 127.11.106.126:39939
I20260619 06:57:59.896214 13007 heartbeater.cc:464] Registering TS with master...
I20260619 06:57:59.896803 13007 heartbeater.cc:511] Master 127.11.106.126:39939 requested a full tablet report, sending...
I20260619 06:57:59.898474 12803 ts_manager.cc:194] Registered new tserver with Master: f98d49dcab6040f9bf121b08c04bf554 (127.11.106.65:32789)
I20260619 06:57:59.899911 12803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:49197
I20260619 06:57:59.901255 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 12876
I20260619 06:57:59.901376 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:57:59.901661 11689 external_mini_cluster.cc:1428] Setting key fda47415c06de2363e2e1940f92fe6ed
I20260619 06:57:59.906298 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:59.898339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39939 (local address 127.11.106.65:48521)
0619 06:57:59.898689 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 06:57:59.898704 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:57:59.898803 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:57:59.899102 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:57:59.899105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:57:59.899331 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:57:59.899443 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:59.899451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.900319 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.900323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:59.901052 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:57:59.901067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.901787 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.901791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:57:59.901896 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:59.902485 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:57:59.902500 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:57:59.902937 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:57:59.904835 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:59.904841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:59.904845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:59.905094 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:59.905389 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:57:59.905391 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:57:59.905392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:57:59.905466 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:57:59.905883 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:57:59.905885 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:57:59.905958 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 06:57:59.906037 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:57:59.906137 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
I20260619 06:57:59.906376 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:57:59.898492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:48521 (local address 127.11.106.126:39939)
0619 06:57:59.898618 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:57:59.898622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:57:59.898795 (+ 173us) server_negotiation.cc:407] Connection header received
0619 06:57:59.898937 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:57:59.898940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:57:59.898984 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:57:59.899066 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:57:59.899572 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.900170 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.901190 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:57:59.901534 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:57:59.901655 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:57:59.903122 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:57:59.903129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:57:59.903136 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:57:59.903165 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:57:59.904704 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:59.905188 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:59.905193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:59.905196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:59.905292 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:57:59.905552 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:57:59.905557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:57:59.905560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:57:59.905789 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:57:59.905901 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:57:59.906131 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 06:57:59.906242 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:57:59.931581 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39939
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:00.047408 13016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:00.047699 13016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:00.047749 13016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:00.052191 13016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:00.052278 13016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:00.052320 13016 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:58:00.052356 13016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:00.058064 13016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.106.126:39939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:00.059244 13016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:00.060341 13016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:00.071054 13021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:00.071066 13024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:00.071137 13016 server_base.cc:1034] running on GCE node
W20260619 06:58:00.071054 13022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:00.071727 13016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:00.072423 13016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:00.073594 13016 hybrid_clock.cc:648] HybridClock initialized: now 1781852280073566 us; error 38 us; skew 500 ppm
Jun 19 06:58:00 dist-test-slave-10d0 krb5kdc[12759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:00.076452 13016 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:58:00.077432 13016 webserver.cc:460] Webserver started at http://127.11.106.66:37129/ using document root <none> and password file <none>
I20260619 06:58:00.078032 13016 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:00.078094 13016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:00.078336 13016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:00.080436 13016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "7ed9c17921614be587c90e798a9a3f3d"
format_stamp: "Formatted at 2026-06-19 06:58:00 on dist-test-slave-10d0"
server_key: "70331f7c088ac2f4fd298a99a1928e15"
server_key_iv: "474a6d0679c93bd888e25e287a092388"
server_key_version: "encryptionkey@0"
I20260619 06:58:00.081090 13016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "7ed9c17921614be587c90e798a9a3f3d"
format_stamp: "Formatted at 2026-06-19 06:58:00 on dist-test-slave-10d0"
server_key: "70331f7c088ac2f4fd298a99a1928e15"
server_key_iv: "474a6d0679c93bd888e25e287a092388"
server_key_version: "encryptionkey@0"
I20260619 06:58:00.085210 13016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:58:00.087795 13031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:00.089241 13016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:00.089404 13016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "7ed9c17921614be587c90e798a9a3f3d"
format_stamp: "Formatted at 2026-06-19 06:58:00 on dist-test-slave-10d0"
server_key: "70331f7c088ac2f4fd298a99a1928e15"
server_key_iv: "474a6d0679c93bd888e25e287a092388"
server_key_version: "encryptionkey@0"
I20260619 06:58:00.089525 13016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:00.102787 13016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:00.105772 13016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:00.105986 13016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:00.107390 13016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:00.107465 13016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:00.107537 13016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:00.107587 13016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:58:00 dist-test-slave-10d0 krb5kdc[12759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:00.118947 13016 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:35505
I20260619 06:58:00.119027 13146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:35505 every 8 connection(s)
I20260619 06:58:00.120132 13016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:00.122287 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.108474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:38669 (local address 127.11.106.126:39939)
0619 06:58:00.108629 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.108632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.109686 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 06:58:00.110724 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.110727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.110822 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.110918 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:00.112627 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.113258 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.114404 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.114781 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.114904 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.118094 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:00.118098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:00.118101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:00.118133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:00.119930 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.120517 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.120523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.120527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.120665 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.121096 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.121100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.121102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.121325 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:00.121427 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.121958 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.122091 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:58:00.123188 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.109041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39939 (local address 127.11.106.66:38669)
0619 06:58:00.109537 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.109579 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.110509 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.111152 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.111174 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.111655 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:00.112408 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.112432 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.113378 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.113382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.114276 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.114285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.115097 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.115101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.115302 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.115896 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:00.115918 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:00.117914 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:00.120047 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.120052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.120064 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.120382 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.120781 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.120788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.120791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.120969 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.121423 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:00.121430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:00.121703 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.121900 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.122287 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260619 06:58:00.124629 13147 heartbeater.cc:347] Connected to a master server at 127.11.106.126:39939
I20260619 06:58:00.124857 13147 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:00.125360 13147 heartbeater.cc:511] Master 127.11.106.126:39939 requested a full tablet report, sending...
I20260619 06:58:00.126466 12804 ts_manager.cc:194] Registered new tserver with Master: 7ed9c17921614be587c90e798a9a3f3d (127.11.106.66:35505)
I20260619 06:58:00.127192 12804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:38669
I20260619 06:58:00.128506 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 13016
I20260619 06:58:00.128634 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:00.128906 11689 external_mini_cluster.cc:1428] Setting key 5a19355622a0e8ded703a0b38bb8a43f
I20260619 06:58:00.135242 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.126709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39939 (local address 127.11.106.66:41523)
0619 06:58:00.127205 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.127219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.127338 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.127691 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.127694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.127913 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:00.128012 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.128017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.129074 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.129079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.129867 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.129877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.130393 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.130397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.130512 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.131078 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:00.131090 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:00.131429 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:00.133503 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.133506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.133508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.133701 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.134093 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.134096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.134097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.134193 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.134796 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:00.134798 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:00.134877 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.134965 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.135079 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":101,"threads_started":1}
I20260619 06:58:00.135455 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.126806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:41523 (local address 127.11.106.126:39939)
0619 06:58:00.126925 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.126930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.127298 (+ 368us) server_negotiation.cc:407] Connection header received
0619 06:58:00.127445 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.127448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.127496 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.127579 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:00.128127 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.128891 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.130009 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.130270 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.130393 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.131584 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:00.131589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:00.131592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:00.131621 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:00.133387 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.133806 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.133813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.133818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.133925 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.134310 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.134317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.134322 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.134670 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:00.134805 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.135069 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.135219 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:58:00.159384 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39939
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:00.278549 13156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:00.278837 13156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:00.278884 13156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:00.283006 13156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:00.283070 13156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:00.283111 13156 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:58:00.283144 13156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:00.288468 13156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.106.126:39939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:00.289554 13156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:00.290547 13156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:00.300658 13164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:00.300823 13156 server_base.cc:1034] running on GCE node
W20260619 06:58:00.300745 13162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:00.300689 13161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:00.301438 13156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:00.302158 13156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:00.303370 13156 hybrid_clock.cc:648] HybridClock initialized: now 1781852280303326 us; error 58 us; skew 500 ppm
Jun 19 06:58:00 dist-test-slave-10d0 krb5kdc[12759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:00.306398 13156 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:58:00.307356 13156 webserver.cc:460] Webserver started at http://127.11.106.67:34567/ using document root <none> and password file <none>
I20260619 06:58:00.307963 13156 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:00.308058 13156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:00.308317 13156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:00.310580 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "12f82e3c015849bba461e83fb60207d9"
format_stamp: "Formatted at 2026-06-19 06:58:00 on dist-test-slave-10d0"
server_key: "61ee9f126580d894954865a291dd3254"
server_key_iv: "2bf672cad3acfb12a43e8ed9d3f80de2"
server_key_version: "encryptionkey@0"
I20260619 06:58:00.311187 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "12f82e3c015849bba461e83fb60207d9"
format_stamp: "Formatted at 2026-06-19 06:58:00 on dist-test-slave-10d0"
server_key: "61ee9f126580d894954865a291dd3254"
server_key_iv: "2bf672cad3acfb12a43e8ed9d3f80de2"
server_key_version: "encryptionkey@0"
I20260619 06:58:00.315143 13156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 06:58:00.317973 13171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:00.319402 13156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:58:00.319569 13156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "12f82e3c015849bba461e83fb60207d9"
format_stamp: "Formatted at 2026-06-19 06:58:00 on dist-test-slave-10d0"
server_key: "61ee9f126580d894954865a291dd3254"
server_key_iv: "2bf672cad3acfb12a43e8ed9d3f80de2"
server_key_version: "encryptionkey@0"
I20260619 06:58:00.319702 13156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:00.352244 13156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:00.355710 13156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:00.355880 13156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:00.357421 13156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:00.357481 13156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:00.357559 13156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:00.357592 13156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:58:00 dist-test-slave-10d0 krb5kdc[12759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852280, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:00.368402 13156 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:40551
I20260619 06:58:00.368453 13286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:40551 every 8 connection(s)
I20260619 06:58:00.369517 13156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:00.373090 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.358389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:43751 (local address 127.11.106.126:39939)
0619 06:58:00.358521 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.358525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.359750 (+ 1225us) server_negotiation.cc:407] Connection header received
0619 06:58:00.361076 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.361081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.361158 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.361269 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:00.362837 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.363723 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.365510 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.365788 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.365910 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.369146 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:00.369149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:00.369152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:00.369181 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:00.370992 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.371479 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.371482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.371483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.371562 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.371974 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.371977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.371979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.372176 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:00.372274 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.372857 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.372970 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:58:00.373924 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.358992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39939 (local address 127.11.106.67:43751)
0619 06:58:00.359591 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.359635 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.360805 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.361488 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.361497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.361939 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:00.362674 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.362698 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.363856 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.363861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.365391 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.365403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.365963 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.365967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.366195 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.366918 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:00.366944 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:00.368955 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:00.371105 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.371109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.371121 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.371380 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.371676 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.371679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.371680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.371841 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.372271 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:00.372275 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:00.372505 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.372758 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.373037 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":145,"threads_started":1}
I20260619 06:58:00.375161 13287 heartbeater.cc:347] Connected to a master server at 127.11.106.126:39939
I20260619 06:58:00.375334 13287 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:00.375852 13287 heartbeater.cc:511] Master 127.11.106.126:39939 requested a full tablet report, sending...
I20260619 06:58:00.376781 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 13156
I20260619 06:58:00.376911 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:00.377005 12803 ts_manager.cc:194] Registered new tserver with Master: 12f82e3c015849bba461e83fb60207d9 (127.11.106.67:40551)
I20260619 06:58:00.377205 11689 external_mini_cluster.cc:1428] Setting key 4bc4b5384faaf2bebf624f88bbf7187e
I20260619 06:58:00.377748 12803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:43751
I20260619 06:58:00.379818 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:58:00.385517 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.377757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:50135 (local address 127.11.106.126:39939)
0619 06:58:00.377882 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.377886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.377898 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:58:00.377973 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.377976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.378027 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.378104 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:00.378696 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.379429 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.380370 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.380781 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.380914 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.382243 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:00.382246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:00.382248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:00.382270 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:00.383856 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.384274 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.384279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.384280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.384364 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.384683 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.384686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.384688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.384867 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:00.384967 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.385248 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.385359 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:58:00.385555 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.377496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39939 (local address 127.11.106.67:50135)
0619 06:58:00.377791 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.377806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.377893 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.378118 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.378120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.378338 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:00.378560 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.378569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.379569 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.379572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.380251 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.380260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.380926 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.380929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.381006 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.381569 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:00.381588 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:00.382108 (+ 520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:00.383966 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.383971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.383974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.384174 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.384452 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.384456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.384458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.384590 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.384968 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:00.384971 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:00.385076 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.385189 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.385355 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
Jun 19 06:58:00 dist-test-slave-10d0 krb5kdc[12759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:58:00 dist-test-slave-10d0 krb5kdc[12759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:00.406599 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.398684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.11.106.126:39939)
0619 06:58:00.398826 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.398829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.398843 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:58:00.398884 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.398886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.398924 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.398994 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:00.399664 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.400185 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.401063 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.401276 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.401355 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.403629 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:00.403632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:00.403635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:00.403660 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:00.405147 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.405530 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.405533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.405534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.405605 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.405842 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.405844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.405845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.406007 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:00.406100 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.406297 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.406391 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:58:00.409621 12804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:00.412734 12804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:00.428043 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.421665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:32789 (local address 127.0.0.1:38504)
0619 06:58:00.422549 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.422600 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.422722 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.423307 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.423313 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.423339 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:00.423499 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.423507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.424912 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.424916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.426400 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.426412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.427435 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.427439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.427590 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.427652 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.427773 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.427893 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":102,"thread_start_us":85,"threads_started":1}
I20260619 06:58:00.428043 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.421816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.11.106.65:32789)
0619 06:58:00.422705 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.422712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.422738 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:58:00.422933 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.422939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.423164 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.423317 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:00.423640 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.424802 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.426538 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.427319 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.427434 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.427580 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.427803 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.427895 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":98,"threads_started":1}
I20260619 06:58:00.428090 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.422007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.11.106.66:35505)
0619 06:58:00.422491 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.422507 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.423092 (+ 585us) server_negotiation.cc:407] Connection header received
0619 06:58:00.423254 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.423263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.423466 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.423575 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:00.424086 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.425212 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.426519 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.427073 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.427197 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.427379 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.427823 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.427922 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":105,"threads_started":1}
I20260619 06:58:00.428687 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.422583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:40551 (local address 127.0.0.1:33080)
0619 06:58:00.423521 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.423535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.423628 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.424085 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.424088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.424107 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:00.424200 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.424206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.425955 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.425959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.427050 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.427061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.428273 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.428276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.428375 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.428401 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.428475 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.428523 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":79,"threads_started":1}
I20260619 06:58:00.429221 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.422965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.11.106.67:40551)
0619 06:58:00.423433 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.423444 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.423628 (+ 184us) server_negotiation.cc:407] Connection header received
0619 06:58:00.423736 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.423740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.423928 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.424113 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:00.424408 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.425838 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.427183 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.427963 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.428058 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.428208 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.429001 (+ 793us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.429091 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":93,"threads_started":1}
I20260619 06:58:00.428687 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.421848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:35505 (local address 127.0.0.1:37398)
0619 06:58:00.422993 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.423003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.423104 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.423773 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.423775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.423792 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:00.423922 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.423928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.425343 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.425346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.426401 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.426410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.427442 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.427445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.427605 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.427650 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.428442 (+ 792us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.428519 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"mutex_wait_us":78,"thread_start_us":91,"threads_started":1}
I20260619 06:58:00.431337 12941 tablet_service.cc:1467] Processing CreateTablet for tablet f0b76147ea0c42588e60c953e81602a9 (DEFAULT_TABLE table=test-table [id=4e9d421328f2451e8b0d41700c589e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:00.431710 13081 tablet_service.cc:1467] Processing CreateTablet for tablet f0b76147ea0c42588e60c953e81602a9 (DEFAULT_TABLE table=test-table [id=4e9d421328f2451e8b0d41700c589e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:00.432415 12941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b76147ea0c42588e60c953e81602a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:00.432772 13081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b76147ea0c42588e60c953e81602a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:00.435144 13221 tablet_service.cc:1467] Processing CreateTablet for tablet f0b76147ea0c42588e60c953e81602a9 (DEFAULT_TABLE table=test-table [id=4e9d421328f2451e8b0d41700c589e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:00.436156 13221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b76147ea0c42588e60c953e81602a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:00.439405 13310 tablet_bootstrap.cc:492] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: Bootstrap starting.
I20260619 06:58:00.440716 13311 tablet_bootstrap.cc:492] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: Bootstrap starting.
I20260619 06:58:00.441658 13310 tablet_bootstrap.cc:654] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:00.442395 13310 log.cc:824] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:00.442787 13311 tablet_bootstrap.cc:654] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:00.443583 13311 log.cc:824] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:00.444141 13313 tablet_bootstrap.cc:492] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: Bootstrap starting.
I20260619 06:58:00.444944 13310 tablet_bootstrap.cc:492] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: No bootstrap required, opened a new log
I20260619 06:58:00.445143 13310 ts_tablet_manager.cc:1397] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:58:00.445488 13311 tablet_bootstrap.cc:492] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: No bootstrap required, opened a new log
I20260619 06:58:00.445694 13311 ts_tablet_manager.cc:1397] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:58:00.446558 13313 tablet_bootstrap.cc:654] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:00.447400 13313 log.cc:824] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:00.449126 13310 raft_consensus.cc:348] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.449407 13310 raft_consensus.cc:374] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:00.449478 13310 raft_consensus.cc:729] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed9c17921614be587c90e798a9a3f3d, State: Initialized, Role: FOLLOWER
I20260619 06:58:00.449501 13313 tablet_bootstrap.cc:492] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: No bootstrap required, opened a new log
I20260619 06:58:00.449672 13313 ts_tablet_manager.cc:1397] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:58:00.449568 13311 raft_consensus.cc:348] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.449848 13311 raft_consensus.cc:374] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:00.449903 13311 raft_consensus.cc:729] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98d49dcab6040f9bf121b08c04bf554, State: Initialized, Role: FOLLOWER
I20260619 06:58:00.450169 13310 consensus_queue.cc:260] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.450347 13311 consensus_queue.cc:260] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.451174 13311 ts_tablet_manager.cc:1428] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:58:00.451195 13310 ts_tablet_manager.cc:1428] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:58:00.451545 13147 heartbeater.cc:503] Master 127.11.106.126:39939 was elected leader, sending a full tablet report...
I20260619 06:58:00.451539 13007 heartbeater.cc:503] Master 127.11.106.126:39939 was elected leader, sending a full tablet report...
I20260619 06:58:00.454833 13313 raft_consensus.cc:348] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.455173 13313 raft_consensus.cc:374] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:00.455242 13313 raft_consensus.cc:729] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f82e3c015849bba461e83fb60207d9, State: Initialized, Role: FOLLOWER
I20260619 06:58:00.455719 13313 consensus_queue.cc:260] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.456430 13287 heartbeater.cc:503] Master 127.11.106.126:39939 was elected leader, sending a full tablet report...
I20260619 06:58:00.456822 13313 ts_tablet_manager.cc:1428] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 06:58:00.481680 13316 raft_consensus.cc:482] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:00.481949 13316 raft_consensus.cc:504] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.483346 13316 leader_election.cc:288] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f98d49dcab6040f9bf121b08c04bf554 (127.11.106.65:32789), 12f82e3c015849bba461e83fb60207d9 (127.11.106.67:40551)
I20260619 06:58:00.487088 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.483749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:41029 (local address 127.11.106.65:32789)
0619 06:58:00.483880 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.483883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.483898 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:00.484001 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.484004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.484063 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.484166 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:00.484473 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.485050 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.486093 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.486527 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.486658 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.486716 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.486896 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.486972 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:58:00.487088 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.483642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:32789 (local address 127.11.106.66:41029)
0619 06:58:00.483782 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.483799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.483898 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.484184 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.484187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.484208 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:00.484346 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.484352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.485166 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.485171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.485962 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.485971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.486646 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.486650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.486756 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.486777 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.486880 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.486953 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 06:58:00.487761 12961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b76147ea0c42588e60c953e81602a9" candidate_uuid: "7ed9c17921614be587c90e798a9a3f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98d49dcab6040f9bf121b08c04bf554" is_pre_election: true
I20260619 06:58:00.488072 12961 raft_consensus.cc:2454] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed9c17921614be587c90e798a9a3f3d in term 0.
I20260619 06:58:00.488215 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.484131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:46745 (local address 127.11.106.67:40551)
0619 06:58:00.484287 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.484293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.484508 (+ 215us) server_negotiation.cc:407] Connection header received
0619 06:58:00.484661 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.484664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.484705 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.484792 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:00.485066 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.485572 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.487163 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.487670 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.487806 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.487855 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.488028 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.488094 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:58:00.488215 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.483978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:40551 (local address 127.11.106.66:46745)
0619 06:58:00.484414 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.484424 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.484544 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.484803 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.484806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.484820 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:00.484953 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.484958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.485683 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.485689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.487033 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.487044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.487821 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.487825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.487928 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.487948 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.488014 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.488079 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":158,"threads_started":1}
I20260619 06:58:00.488618 13032 leader_election.cc:302] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9c17921614be587c90e798a9a3f3d, f98d49dcab6040f9bf121b08c04bf554; no voters:
I20260619 06:58:00.488831 13241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b76147ea0c42588e60c953e81602a9" candidate_uuid: "7ed9c17921614be587c90e798a9a3f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f82e3c015849bba461e83fb60207d9" is_pre_election: true
I20260619 06:58:00.488934 13316 raft_consensus.cc:2781] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:00.489053 13316 raft_consensus.cc:482] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:00.489105 13316 raft_consensus.cc:3037] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:00.489162 13241 raft_consensus.cc:2454] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed9c17921614be587c90e798a9a3f3d in term 0.
I20260619 06:58:00.490264 13316 raft_consensus.cc:504] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.490597 13316 leader_election.cc:288] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [CANDIDATE]: Term 1 election: Requested vote from peers f98d49dcab6040f9bf121b08c04bf554 (127.11.106.65:32789), 12f82e3c015849bba461e83fb60207d9 (127.11.106.67:40551)
I20260619 06:58:00.490984 12961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b76147ea0c42588e60c953e81602a9" candidate_uuid: "7ed9c17921614be587c90e798a9a3f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98d49dcab6040f9bf121b08c04bf554"
I20260619 06:58:00.491118 13241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b76147ea0c42588e60c953e81602a9" candidate_uuid: "7ed9c17921614be587c90e798a9a3f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f82e3c015849bba461e83fb60207d9"
I20260619 06:58:00.491118 12961 raft_consensus.cc:3037] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:00.491273 13241 raft_consensus.cc:3037] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:00.492199 12961 raft_consensus.cc:2454] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed9c17921614be587c90e798a9a3f3d in term 1.
I20260619 06:58:00.492379 13241 raft_consensus.cc:2454] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed9c17921614be587c90e798a9a3f3d in term 1.
I20260619 06:58:00.492515 13032 leader_election.cc:302] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9c17921614be587c90e798a9a3f3d, f98d49dcab6040f9bf121b08c04bf554; no voters:
I20260619 06:58:00.492765 13316 raft_consensus.cc:2781] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:00.493052 13316 raft_consensus.cc:686] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 1 LEADER]: Becoming Leader. State: Replica: 7ed9c17921614be587c90e798a9a3f3d, State: Running, Role: LEADER
I20260619 06:58:00.493391 13316 consensus_queue.cc:237] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.497720 12804 catalog_manager.cc:5495] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ed9c17921614be587c90e798a9a3f3d (127.11.106.66). New cstate: current_term: 1 leader_uuid: "7ed9c17921614be587c90e798a9a3f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:00.532788 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.528758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.11.106.66:35505)
0619 06:58:00.528918 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.528925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.528942 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:58:00.529195 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.529199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.529273 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.529387 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:00.529667 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.530195 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.531112 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.531358 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.531473 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.531825 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:00.532331 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:00.532467 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.532573 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.532635 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:58:00.539034 12961 raft_consensus.cc:1261] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Refusing update from remote peer 7ed9c17921614be587c90e798a9a3f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:00.539111 13241 raft_consensus.cc:1261] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Refusing update from remote peer 7ed9c17921614be587c90e798a9a3f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:00.539808 13316 consensus_queue.cc:1035] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:00.540024 13320 consensus_queue.cc:1035] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:00.549582 13325 mvcc.cc:205] Tried to move back new op lower bound from 7298466941079224320 to 7298466940903796736. Current Snapshot: MvccSnapshot[applied={T|T < 7298466941079224320}]
I20260619 06:58:00.562886 12803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:00.563090 12803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:00.566053 12803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d: Sending DeleteTablet for 3 replicas of tablet f0b76147ea0c42588e60c953e81602a9
I20260619 06:58:00.566964 13081 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b76147ea0c42588e60c953e81602a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:00 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:37398
I20260619 06:58:00.567011 13221 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b76147ea0c42588e60c953e81602a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:00 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:33080
I20260619 06:58:00.567368 12941 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b76147ea0c42588e60c953e81602a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:00 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:38504
I20260619 06:58:00.568018 13335 tablet_replica.cc:330] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: stopping tablet replica
I20260619 06:58:00.568106 13333 tablet_replica.cc:330] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: stopping tablet replica
I20260619 06:58:00.568408 13335 raft_consensus.cc:2229] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:00.568406 13333 raft_consensus.cc:2229] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:00.568493 12803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:58:00.568764 13335 raft_consensus.cc:2258] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:00.568837 13333 raft_consensus.cc:2258] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 06:58:00.568959 12803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:00.570358 13334 tablet_replica.cc:330] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: stopping tablet replica
I20260619 06:58:00.570415 13333 ts_tablet_manager.cc:1905] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:00.570803 13334 raft_consensus.cc:2229] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:00.571237 13334 raft_consensus.cc:2258] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:00.574258 13334 ts_tablet_manager.cc:1905] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:00.574252 13335 ts_tablet_manager.cc:1905] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:00.575245 12941 tablet_service.cc:1467] Processing CreateTablet for tablet 245348cd5b6d4b68bc0dd97fee2b919a (DEFAULT_TABLE table=test-table [id=9fb85a65b8d040d48847682243835f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:00.575824 13221 tablet_service.cc:1467] Processing CreateTablet for tablet 245348cd5b6d4b68bc0dd97fee2b919a (DEFAULT_TABLE table=test-table [id=9fb85a65b8d040d48847682243835f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:00.576076 13221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245348cd5b6d4b68bc0dd97fee2b919a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:00.578426 13334 ts_tablet_manager.cc:1918] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:00.578519 13334 log.cc:1196] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/f0b76147ea0c42588e60c953e81602a9
I20260619 06:58:00.578563 13313 tablet_bootstrap.cc:492] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: Bootstrap starting.
I20260619 06:58:00.578763 13081 tablet_service.cc:1467] Processing CreateTablet for tablet 245348cd5b6d4b68bc0dd97fee2b919a (DEFAULT_TABLE table=test-table [id=9fb85a65b8d040d48847682243835f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:00.578860 13334 ts_tablet_manager.cc:1939] T f0b76147ea0c42588e60c953e81602a9 P 12f82e3c015849bba461e83fb60207d9: Deleting consensus metadata
I20260619 06:58:00.579273 12941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245348cd5b6d4b68bc0dd97fee2b919a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:00.579682 13313 tablet_bootstrap.cc:654] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:00.580113 12788 catalog_manager.cc:4841] TS 12f82e3c015849bba461e83fb60207d9 (127.11.106.67:40551): tablet f0b76147ea0c42588e60c953e81602a9 (table test-table [id=4e9d421328f2451e8b0d41700c589e1c]) successfully deleted
I20260619 06:58:00.580443 13081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245348cd5b6d4b68bc0dd97fee2b919a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:00.581708 13311 tablet_bootstrap.cc:492] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: Bootstrap starting.
I20260619 06:58:00.582100 13335 ts_tablet_manager.cc:1918] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:00.582238 13335 log.cc:1196] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/f0b76147ea0c42588e60c953e81602a9
I20260619 06:58:00.582612 13335 ts_tablet_manager.cc:1939] T f0b76147ea0c42588e60c953e81602a9 P f98d49dcab6040f9bf121b08c04bf554: Deleting consensus metadata
I20260619 06:58:00.582819 13310 tablet_bootstrap.cc:492] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: Bootstrap starting.
I20260619 06:58:00.582906 13333 ts_tablet_manager.cc:1918] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:00.583015 13333 log.cc:1196] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/f0b76147ea0c42588e60c953e81602a9
I20260619 06:58:00.583361 13333 ts_tablet_manager.cc:1939] T f0b76147ea0c42588e60c953e81602a9 P 7ed9c17921614be587c90e798a9a3f3d: Deleting consensus metadata
I20260619 06:58:00.583639 13311 tablet_bootstrap.cc:654] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:00.583758 13310 tablet_bootstrap.cc:654] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:00.584530 12789 catalog_manager.cc:4841] TS 7ed9c17921614be587c90e798a9a3f3d (127.11.106.66:35505): tablet f0b76147ea0c42588e60c953e81602a9 (table test-table [id=4e9d421328f2451e8b0d41700c589e1c]) successfully deleted
I20260619 06:58:00.584990 13311 tablet_bootstrap.cc:492] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: No bootstrap required, opened a new log
I20260619 06:58:00.585080 13311 ts_tablet_manager.cc:1397] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:00.585114 12788 catalog_manager.cc:4841] TS f98d49dcab6040f9bf121b08c04bf554 (127.11.106.65:32789): tablet f0b76147ea0c42588e60c953e81602a9 (table test-table [id=4e9d421328f2451e8b0d41700c589e1c]) successfully deleted
I20260619 06:58:00.585898 13313 tablet_bootstrap.cc:492] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: No bootstrap required, opened a new log
I20260619 06:58:00.585995 13313 ts_tablet_manager.cc:1397] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 06:58:00.585944 13311 raft_consensus.cc:348] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.586064 13311 raft_consensus.cc:374] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:00.586107 13310 tablet_bootstrap.cc:492] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: No bootstrap required, opened a new log
I20260619 06:58:00.586102 13311 raft_consensus.cc:729] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98d49dcab6040f9bf121b08c04bf554, State: Initialized, Role: FOLLOWER
I20260619 06:58:00.586174 13310 ts_tablet_manager.cc:1397] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:00.586225 13311 consensus_queue.cc:260] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.586498 13311 ts_tablet_manager.cc:1428] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 06:58:00.586437 13313 raft_consensus.cc:348] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.586539 13313 raft_consensus.cc:374] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:00.586573 13313 raft_consensus.cc:729] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f82e3c015849bba461e83fb60207d9, State: Initialized, Role: FOLLOWER
I20260619 06:58:00.586577 13310 raft_consensus.cc:348] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.586673 13310 raft_consensus.cc:374] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:00.586702 13310 raft_consensus.cc:729] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed9c17921614be587c90e798a9a3f3d, State: Initialized, Role: FOLLOWER
I20260619 06:58:00.586694 13313 consensus_queue.cc:260] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.586810 13310 consensus_queue.cc:260] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.586946 13313 ts_tablet_manager.cc:1428] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:00.587031 13310 ts_tablet_manager.cc:1428] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260619 06:58:00.620651 13288 tablet.cc:2369] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:58:00.621327 13148 tablet.cc:2369] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:58:00.643849 13008 tablet.cc:2369] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:00.911073 13328 raft_consensus.cc:482] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:00.911269 13328 raft_consensus.cc:504] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.911768 13328 leader_election.cc:288] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f98d49dcab6040f9bf121b08c04bf554 (127.11.106.65:32789), 12f82e3c015849bba461e83fb60207d9 (127.11.106.67:40551)
I20260619 06:58:00.912256 12961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245348cd5b6d4b68bc0dd97fee2b919a" candidate_uuid: "7ed9c17921614be587c90e798a9a3f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98d49dcab6040f9bf121b08c04bf554" is_pre_election: true
I20260619 06:58:00.912320 13241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245348cd5b6d4b68bc0dd97fee2b919a" candidate_uuid: "7ed9c17921614be587c90e798a9a3f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f82e3c015849bba461e83fb60207d9" is_pre_election: true
I20260619 06:58:00.912416 12961 raft_consensus.cc:2454] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed9c17921614be587c90e798a9a3f3d in term 0.
I20260619 06:58:00.912515 13241 raft_consensus.cc:2454] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed9c17921614be587c90e798a9a3f3d in term 0.
I20260619 06:58:00.912822 13032 leader_election.cc:302] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9c17921614be587c90e798a9a3f3d, f98d49dcab6040f9bf121b08c04bf554; no voters:
I20260619 06:58:00.913038 13328 raft_consensus.cc:2781] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:00.913094 13328 raft_consensus.cc:482] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:00.913121 13328 raft_consensus.cc:3037] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:00.914158 13328 raft_consensus.cc:504] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.914494 13328 leader_election.cc:288] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [CANDIDATE]: Term 1 election: Requested vote from peers f98d49dcab6040f9bf121b08c04bf554 (127.11.106.65:32789), 12f82e3c015849bba461e83fb60207d9 (127.11.106.67:40551)
I20260619 06:58:00.914884 12961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245348cd5b6d4b68bc0dd97fee2b919a" candidate_uuid: "7ed9c17921614be587c90e798a9a3f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98d49dcab6040f9bf121b08c04bf554"
I20260619 06:58:00.914944 13241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245348cd5b6d4b68bc0dd97fee2b919a" candidate_uuid: "7ed9c17921614be587c90e798a9a3f3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f82e3c015849bba461e83fb60207d9"
I20260619 06:58:00.915020 12961 raft_consensus.cc:3037] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:00.915071 13241 raft_consensus.cc:3037] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:00.915979 13241 raft_consensus.cc:2454] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed9c17921614be587c90e798a9a3f3d in term 1.
I20260619 06:58:00.915987 12961 raft_consensus.cc:2454] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed9c17921614be587c90e798a9a3f3d in term 1.
I20260619 06:58:00.916254 13032 leader_election.cc:302] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9c17921614be587c90e798a9a3f3d, f98d49dcab6040f9bf121b08c04bf554; no voters:
I20260619 06:58:00.916424 13328 raft_consensus.cc:2781] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:00.916509 13328 raft_consensus.cc:686] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 1 LEADER]: Becoming Leader. State: Replica: 7ed9c17921614be587c90e798a9a3f3d, State: Running, Role: LEADER
I20260619 06:58:00.916682 13328 consensus_queue.cc:237] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:00.918308 12802 catalog_manager.cc:5495] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ed9c17921614be587c90e798a9a3f3d (127.11.106.66). New cstate: current_term: 1 leader_uuid: "7ed9c17921614be587c90e798a9a3f3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } health_report { overall_health: UNKNOWN } } }
Jun 19 06:58:00 dist-test-slave-10d0 krb5kdc[12759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852279, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:00.986961 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.978296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39939 (local address 127.0.0.1:43712)
0619 06:58:00.978690 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:00.978704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:00.978847 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:00.979096 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:00.979099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:00.979549 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:00.979648 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.979654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.980421 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.980426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.981058 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:00.981066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.981515 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.981518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:00.981597 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.982205 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:00.982217 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:00.983635 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:00.985483 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.985491 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.985497 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.985724 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.985971 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:00.985973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:00.985975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:00.986050 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:00.986391 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:00.986396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:00.986501 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 06:58:00.986589 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:58:00.986781 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":106,"threads_started":1}
I20260619 06:58:00.986979 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:00.978304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.11.106.126:39939)
0619 06:58:00.978637 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:00.978646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:00.978787 (+ 141us) server_negotiation.cc:407] Connection header received
0619 06:58:00.978940 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:00.978943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:00.978988 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:00.979078 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:00.979766 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.980314 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.981175 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:00.981403 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:00.981511 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:00.983804 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:00.983808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:00.983810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:00.983832 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:00.985363 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.985809 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.985812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.985813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.985894 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:00.986124 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:00.986127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:00.986128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:00.986302 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:00.986378 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:00.986670 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:58:00.986842 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260619 06:58:00.991041 12802 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:43712:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:58:00.991796 12802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:00.997264 12941 tablet_service.cc:1467] Processing CreateTablet for tablet 489bca0a947546d4a2ec6e72e82aa0b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0afff1f0a91748cc8837a43872b6b004]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:58:00.997481 13081 tablet_service.cc:1467] Processing CreateTablet for tablet 489bca0a947546d4a2ec6e72e82aa0b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0afff1f0a91748cc8837a43872b6b004]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:58:00.997745 12941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489bca0a947546d4a2ec6e72e82aa0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:00.997905 13081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489bca0a947546d4a2ec6e72e82aa0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:01.000367 13310 tablet_bootstrap.cc:492] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: Bootstrap starting.
I20260619 06:58:01.000377 13311 tablet_bootstrap.cc:492] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: Bootstrap starting.
I20260619 06:58:01.000957 13221 tablet_service.cc:1467] Processing CreateTablet for tablet 489bca0a947546d4a2ec6e72e82aa0b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0afff1f0a91748cc8837a43872b6b004]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:58:01.001339 13221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 489bca0a947546d4a2ec6e72e82aa0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:01.001395 13310 tablet_bootstrap.cc:654] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:01.002545 13310 tablet_bootstrap.cc:492] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: No bootstrap required, opened a new log
I20260619 06:58:01.002650 13310 ts_tablet_manager.cc:1397] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:58:01.003103 13310 raft_consensus.cc:348] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:01.003234 13310 raft_consensus.cc:374] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:01.003279 13310 raft_consensus.cc:729] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed9c17921614be587c90e798a9a3f3d, State: Initialized, Role: FOLLOWER
I20260619 06:58:01.003408 13310 consensus_queue.cc:260] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:01.003721 13310 ts_tablet_manager.cc:1428] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:01.003708 13328 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.003870 13328 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:01.004458 13313 tablet_bootstrap.cc:492] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: Bootstrap starting.
I20260619 06:58:01.005496 13313 tablet_bootstrap.cc:654] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:01.005512 13311 tablet_bootstrap.cc:654] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:01.006853 13311 tablet_bootstrap.cc:492] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: No bootstrap required, opened a new log
I20260619 06:58:01.006958 13311 ts_tablet_manager.cc:1397] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 06:58:01.007493 13311 raft_consensus.cc:348] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:01.007623 13311 raft_consensus.cc:374] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:01.007650 13311 raft_consensus.cc:729] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98d49dcab6040f9bf121b08c04bf554, State: Initialized, Role: FOLLOWER
I20260619 06:58:01.007769 13311 consensus_queue.cc:260] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:01.008026 13311 ts_tablet_manager.cc:1428] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:58:01.007984 13317 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.008158 13317 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:01.008733 13313 tablet_bootstrap.cc:492] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: No bootstrap required, opened a new log
I20260619 06:58:01.008850 13313 ts_tablet_manager.cc:1397] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:58:01.009243 13313 raft_consensus.cc:348] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:01.009366 13313 raft_consensus.cc:374] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:01.009449 13313 raft_consensus.cc:729] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f82e3c015849bba461e83fb60207d9, State: Initialized, Role: FOLLOWER
I20260619 06:58:01.009591 13313 consensus_queue.cc:260] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:01.009914 13313 ts_tablet_manager.cc:1428] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:58:01.009860 13318 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.010000 13318 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:01.049199 13317 raft_consensus.cc:482] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:01.049363 13317 raft_consensus.cc:504] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:01.050534 13317 leader_election.cc:288] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ed9c17921614be587c90e798a9a3f3d (127.11.106.66:35505), 12f82e3c015849bba461e83fb60207d9 (127.11.106.67:40551)
I20260619 06:58:01.055599 13353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.051125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:46445 (local address 127.11.106.67:40551)
0619 06:58:01.051386 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:01.051390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:01.051401 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:58:01.051443 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:01.051445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:01.051482 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:01.051551 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:01.051848 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.052618 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.054355 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.055010 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.055111 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.055154 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:01.055373 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 06:58:01.055463 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
I20260619 06:58:01.055603 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.050798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:40551 (local address 127.11.106.65:46445)
0619 06:58:01.051029 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:01.051044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:01.051136 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:01.051580 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:01.051586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:01.051606 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:01.051738 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.051743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.052768 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.052771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.054220 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.054232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.055113 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.055117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.055206 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.055230 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:58:01.055338 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:58:01.055406 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":91,"thread_start_us":64,"threads_started":1}
I20260619 06:58:01.055737 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.050834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:40289 (local address 127.11.106.66:35505)
0619 06:58:01.051141 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:01.051149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:01.051177 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:58:01.051230 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:01.051233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:01.051283 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:01.051373 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:01.051680 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.052508 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.054355 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.054971 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.055100 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.055154 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:01.055512 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 06:58:01.055594 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":57,"threads_started":1}
I20260619 06:58:01.056155 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.050706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:35505 (local address 127.11.106.65:40289)
0619 06:58:01.050974 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:01.050992 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:01.051108 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:01.051385 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:01.051388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:01.051404 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:01.051550 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.051558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.052628 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.052631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.054224 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.054232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.055223 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.055226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.055375 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.055401 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:58:01.055477 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:58:01.055546 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":134,"thread_start_us":101,"threads_started":1}
I20260619 06:58:01.056167 13241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489bca0a947546d4a2ec6e72e82aa0b4" candidate_uuid: "f98d49dcab6040f9bf121b08c04bf554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f82e3c015849bba461e83fb60207d9" is_pre_election: true
I20260619 06:58:01.056310 13241 raft_consensus.cc:2454] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f98d49dcab6040f9bf121b08c04bf554 in term 0.
I20260619 06:58:01.056705 12892 leader_election.cc:302] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f82e3c015849bba461e83fb60207d9, f98d49dcab6040f9bf121b08c04bf554; no voters:
I20260619 06:58:01.056730 13086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489bca0a947546d4a2ec6e72e82aa0b4" candidate_uuid: "f98d49dcab6040f9bf121b08c04bf554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9c17921614be587c90e798a9a3f3d" is_pre_election: true
I20260619 06:58:01.056957 13086 raft_consensus.cc:2454] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f98d49dcab6040f9bf121b08c04bf554 in term 0.
I20260619 06:58:01.056982 13317 raft_consensus.cc:2781] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:01.057058 13317 raft_consensus.cc:482] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:01.057114 13317 raft_consensus.cc:3037] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:01.058117 13317 raft_consensus.cc:504] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:01.058492 13317 leader_election.cc:288] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [CANDIDATE]: Term 1 election: Requested vote from peers 7ed9c17921614be587c90e798a9a3f3d (127.11.106.66:35505), 12f82e3c015849bba461e83fb60207d9 (127.11.106.67:40551)
I20260619 06:58:01.058845 13241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489bca0a947546d4a2ec6e72e82aa0b4" candidate_uuid: "f98d49dcab6040f9bf121b08c04bf554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f82e3c015849bba461e83fb60207d9"
I20260619 06:58:01.058959 13241 raft_consensus.cc:3037] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:01.059839 13241 raft_consensus.cc:2454] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f98d49dcab6040f9bf121b08c04bf554 in term 1.
I20260619 06:58:01.060098 12892 leader_election.cc:302] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f82e3c015849bba461e83fb60207d9, f98d49dcab6040f9bf121b08c04bf554; no voters:
I20260619 06:58:01.060091 13086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "489bca0a947546d4a2ec6e72e82aa0b4" candidate_uuid: "f98d49dcab6040f9bf121b08c04bf554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9c17921614be587c90e798a9a3f3d"
I20260619 06:58:01.060202 13086 raft_consensus.cc:3037] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:01.060297 13317 raft_consensus.cc:2781] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:01.060513 13317 raft_consensus.cc:686] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [term 1 LEADER]: Becoming Leader. State: Replica: f98d49dcab6040f9bf121b08c04bf554, State: Running, Role: LEADER
I20260619 06:58:01.060791 13317 consensus_queue.cc:237] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } }
I20260619 06:58:01.061439 13086 raft_consensus.cc:2454] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f98d49dcab6040f9bf121b08c04bf554 in term 1.
I20260619 06:58:01.062860 13354 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: TxnStatusTablet state changed. Reason: New leader f98d49dcab6040f9bf121b08c04bf554. Latest consensus state: current_term: 1 leader_uuid: "f98d49dcab6040f9bf121b08c04bf554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.062978 13354 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:58:01.063247 13356 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:58:01.063431 13356 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:58:01.063454 12802 catalog_manager.cc:5495] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 reported cstate change: term changed from 0 to 1, leader changed from <none> to f98d49dcab6040f9bf121b08c04bf554 (127.11.106.65). New cstate: current_term: 1 leader_uuid: "f98d49dcab6040f9bf121b08c04bf554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:01.087919 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.083266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:32789 (local address 127.0.0.1:38514)
0619 06:58:01.083408 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:01.083425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:01.083544 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:01.083985 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:01.083990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:01.083998 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:01.084115 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.084125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.085255 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.085258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.086060 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.086070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.086675 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.086677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.086756 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.086807 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:01.087621 (+ 814us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:01.087642 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:58:01.087718 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:58:01.087802 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 06:58:01.087976 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.083372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.11.106.65:32789)
0619 06:58:01.083700 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:01.083708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:01.083720 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:58:01.083786 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:01.083788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:01.083841 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:01.083958 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:01.084274 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.085129 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.086173 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.086554 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.086656 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.086951 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:01.087515 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:01.087635 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:01.087751 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 06:58:01.087815 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":134,"threads_started":1}
I20260619 06:58:01.091066 13241 raft_consensus.cc:1261] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Refusing update from remote peer f98d49dcab6040f9bf121b08c04bf554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:01.091210 13086 raft_consensus.cc:1261] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d [term 1 FOLLOWER]: Refusing update from remote peer f98d49dcab6040f9bf121b08c04bf554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:01.091527 13354 consensus_queue.cc:1035] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:01.091745 13317 consensus_queue.cc:1035] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:01.093348 13318 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: TxnStatusTablet state changed. Reason: New leader f98d49dcab6040f9bf121b08c04bf554. Latest consensus state: current_term: 1 leader_uuid: "f98d49dcab6040f9bf121b08c04bf554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.093457 13318 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:01.094856 13328 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: TxnStatusTablet state changed. Reason: New leader f98d49dcab6040f9bf121b08c04bf554. Latest consensus state: current_term: 1 leader_uuid: "f98d49dcab6040f9bf121b08c04bf554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.094975 13328 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:01.097221 13317 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f98d49dcab6040f9bf121b08c04bf554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.097342 13317 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:58:01.097446 13360 mvcc.cc:205] Tried to move back new op lower bound from 7298466943343558656 to 7298466943227936768. Current Snapshot: MvccSnapshot[applied={T|T < 7298466943343558656 or (T in {7298466943343558656})}]
I20260619 06:58:01.101893 13318 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f98d49dcab6040f9bf121b08c04bf554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.102028 13318 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P 12f82e3c015849bba461e83fb60207d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:01.102564 13328 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f98d49dcab6040f9bf121b08c04bf554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.102705 13328 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P 7ed9c17921614be587c90e798a9a3f3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:01.103988 13317 tablet_replica.cc:439] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f98d49dcab6040f9bf121b08c04bf554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 } } peers { permanent_uuid: "7ed9c17921614be587c90e798a9a3f3d" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 35505 } } peers { permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 } } }
I20260619 06:58:01.104104 13317 tablet_replica.cc:442] T 489bca0a947546d4a2ec6e72e82aa0b4 P f98d49dcab6040f9bf121b08c04bf554: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:58:01.104326 13363 mvcc.cc:205] Tried to move back new op lower bound from 7298466943343558656 to 7298466943227936768. Current Snapshot: MvccSnapshot[applied={T|T < 7298466943343558656 or (T in {7298466943343558656})}]
I20260619 06:58:01.113602 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.109848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:41353 (local address 127.11.106.65:32789)
0619 06:58:01.110014 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:01.110019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:01.110081 (+ 62us) server_negotiation.cc:407] Connection header received
0619 06:58:01.110227 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:01.110230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:01.110300 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:01.110384 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:01.110591 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.111372 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.112306 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.112686 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.112811 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.113133 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:01.113219 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:01.113285 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:01.113407 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 06:58:01.113462 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:58:01.113610 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.109675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:32789 (local address 127.11.106.66:41353)
0619 06:58:01.110016 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:01.110029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:01.110123 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:01.110401 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:01.110403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:01.110410 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:01.110495 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.110500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.111558 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.111561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.112191 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.112200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.112830 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.112834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.112916 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.112974 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:01.113310 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:01.113316 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:01.113382 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 06:58:01.113465 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
I20260619 06:58:01.120781 13241 raft_consensus.cc:1261] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Refusing update from remote peer 7ed9c17921614be587c90e798a9a3f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:01.120843 12961 raft_consensus.cc:1261] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Refusing update from remote peer 7ed9c17921614be587c90e798a9a3f3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:01.121274 13316 consensus_queue.cc:1035] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f98d49dcab6040f9bf121b08c04bf554" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:01.121479 13328 consensus_queue.cc:1035] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f82e3c015849bba461e83fb60207d9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:01.152016 13352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.147983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:42213 (local address 127.11.106.66:35505)
0619 06:58:01.148116 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:01.148121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:01.148347 (+ 226us) server_negotiation.cc:407] Connection header received
0619 06:58:01.148444 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:01.148447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:01.148500 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:01.148618 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:01.148856 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.149674 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.150956 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.151183 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.151270 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.151614 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:01.151682 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:01.151743 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:01.151857 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 06:58:01.151909 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:58:01.152089 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.147874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:35505 (local address 127.11.106.65:42213)
0619 06:58:01.148266 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:01.148279 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:01.148370 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:01.148626 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:01.148628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:01.148636 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:01.148759 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.148764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.149817 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.149821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.150835 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:01.150847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.151325 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.151329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:01.151426 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.151484 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:01.151764 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:01.151771 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:58:01.151839 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:58:01.151923 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":97,"threads_started":1}
I20260619 06:58:01.177858 12803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:01.178036 12803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:01.180209 12803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e2f95f46ce44b8bb27405b0c6688f4d: Sending DeleteTablet for 3 replicas of tablet 245348cd5b6d4b68bc0dd97fee2b919a
I20260619 06:58:01.180859 13081 tablet_service.cc:1514] Processing DeleteTablet for tablet 245348cd5b6d4b68bc0dd97fee2b919a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:01 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:37398
I20260619 06:58:01.181218 13221 tablet_service.cc:1514] Processing DeleteTablet for tablet 245348cd5b6d4b68bc0dd97fee2b919a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:01 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:33080
I20260619 06:58:01.181263 13379 tablet_replica.cc:330] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: stopping tablet replica
I20260619 06:58:01.181547 13379 raft_consensus.cc:2229] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:01.181576 13380 tablet_replica.cc:330] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: stopping tablet replica
I20260619 06:58:01.181703 13380 raft_consensus.cc:2229] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:01.181859 13380 raft_consensus.cc:2258] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:01.181867 13379 raft_consensus.cc:2258] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:01.181900 12941 tablet_service.cc:1514] Processing DeleteTablet for tablet 245348cd5b6d4b68bc0dd97fee2b919a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:01 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:38504
I20260619 06:58:01.182288 13382 tablet_replica.cc:330] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: stopping tablet replica
I20260619 06:58:01.182416 13382 raft_consensus.cc:2229] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:01.182540 13382 raft_consensus.cc:2258] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:01.182600 13380 ts_tablet_manager.cc:1905] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:01.182981 13379 ts_tablet_manager.cc:1905] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:01.183928 13382 ts_tablet_manager.cc:1905] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:01.186110 13379 ts_tablet_manager.cc:1918] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:01.186250 13379 log.cc:1196] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/245348cd5b6d4b68bc0dd97fee2b919a
I20260619 06:58:01.186275 13380 ts_tablet_manager.cc:1918] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:01.186354 13380 log.cc:1196] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/245348cd5b6d4b68bc0dd97fee2b919a
I20260619 06:58:01.186625 13380 ts_tablet_manager.cc:1939] T 245348cd5b6d4b68bc0dd97fee2b919a P 12f82e3c015849bba461e83fb60207d9: Deleting consensus metadata
I20260619 06:58:01.186759 13379 ts_tablet_manager.cc:1939] T 245348cd5b6d4b68bc0dd97fee2b919a P 7ed9c17921614be587c90e798a9a3f3d: Deleting consensus metadata
I20260619 06:58:01.187107 13382 ts_tablet_manager.cc:1918] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:01.187192 13382 log.cc:1196] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/245348cd5b6d4b68bc0dd97fee2b919a
I20260619 06:58:01.187461 13382 ts_tablet_manager.cc:1939] T 245348cd5b6d4b68bc0dd97fee2b919a P f98d49dcab6040f9bf121b08c04bf554: Deleting consensus metadata
I20260619 06:58:01.187485 12788 catalog_manager.cc:4841] TS 12f82e3c015849bba461e83fb60207d9 (127.11.106.67:40551): tablet 245348cd5b6d4b68bc0dd97fee2b919a (table test-table [id=9fb85a65b8d040d48847682243835f11]) successfully deleted
I20260619 06:58:01.187668 12789 catalog_manager.cc:4841] TS 7ed9c17921614be587c90e798a9a3f3d (127.11.106.66:35505): tablet 245348cd5b6d4b68bc0dd97fee2b919a (table test-table [id=9fb85a65b8d040d48847682243835f11]) successfully deleted
I20260619 06:58:01.188208 12788 catalog_manager.cc:4841] TS f98d49dcab6040f9bf121b08c04bf554 (127.11.106.65:32789): tablet 245348cd5b6d4b68bc0dd97fee2b919a (table test-table [id=9fb85a65b8d040d48847682243835f11]) successfully deleted
Jun 19 06:58:01 dist-test-slave-10d0 krb5kdc[12759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:58:01.198096 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.182977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.11.106.65:32789)
0619 06:58:01.183082 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:01.183086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:01.187981 (+ 4895us) server_negotiation.cc:407] Connection header received
0619 06:58:01.188239 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:01.188242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:01.188299 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:01.188356 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:01.189206 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.189728 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.190603 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.190834 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.190950 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.193664 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:01.193725 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:01.193745 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:01.193776 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:01.196025 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:01.196603 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:01.196610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:01.196612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:01.196724 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:01.197003 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:01.197006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:01.197008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:01.197347 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:01.197578 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:01.197731 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 06:58:01.197886 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 06:58:01.198668 12983 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38526
I20260619 06:58:01.208276 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:01.200586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.11.106.126:39939)
0619 06:58:01.200744 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:01.200751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:01.200769 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:58:01.200935 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:01.200939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:01.200996 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:01.201092 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:01.201735 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.202272 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.203070 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:01.203316 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:01.203430 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:01.204870 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:01.204873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:01.204878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:01.204909 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:01.206463 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:01.206927 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:01.206932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:01.206933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:01.207022 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:01.207314 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:01.207320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:01.207321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:01.207521 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:01.207666 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:01.207964 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 06:58:01.208104 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 06:58:01.208858 12803 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43728
I20260619 06:58:01.210062 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 12876
I20260619 06:58:01.217862 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 13016
I20260619 06:58:01.224951 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 13156
I20260619 06:58:01.230773 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 12772
2026-06-19T06:58:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3013 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:01 dist-test-slave-10d0 krb5kdc[13388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:01 dist-test-slave-10d0 krb5kdc[13388](info): set up 2 sockets
Jun 19 06:58:01 dist-test-slave-10d0 krb5kdc[13388](info): commencing operation
krb5kdc: starting...
W20260619 06:58:02.285117 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:02 dist-test-slave-10d0 krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:02.409622 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:39181
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:39181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:02.524888 13401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:02.525298 13401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:02.525388 13401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:02.529520 13401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:02.529603 13401 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:58:02.529634 13401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:02.529681 13401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:02.529719 13401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:02.529753 13401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:02.535110 13401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:39181
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:39181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:02.536296 13401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:02.537458 13401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:02.546433 13409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:02.546433 13406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:02.546478 13407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:02.546509 13401 server_base.cc:1034] running on GCE node
I20260619 06:58:02.547230 13401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:02.548367 13401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:02.549544 13401 hybrid_clock.cc:648] HybridClock initialized: now 1781852282549528 us; error 32 us; skew 500 ppm
Jun 19 06:58:02 dist-test-slave-10d0 krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852282, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:02.552332 13401 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:02.553274 13401 webserver.cc:460] Webserver started at http://127.11.106.126:39447/ using document root <none> and password file <none>
I20260619 06:58:02.553874 13401 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:02.553934 13401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:02.554170 13401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:02.556221 13401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "fe605bc65bb44a58b6ed04de9790dfaa"
format_stamp: "Formatted at 2026-06-19 06:58:02 on dist-test-slave-10d0"
server_key: "46797b0964c9d265f91878ea49dd7635"
server_key_iv: "f559e1d0f5336d03daf1ccebca3a8c72"
server_key_version: "encryptionkey@0"
I20260619 06:58:02.556793 13401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "fe605bc65bb44a58b6ed04de9790dfaa"
format_stamp: "Formatted at 2026-06-19 06:58:02 on dist-test-slave-10d0"
server_key: "46797b0964c9d265f91878ea49dd7635"
server_key_iv: "f559e1d0f5336d03daf1ccebca3a8c72"
server_key_version: "encryptionkey@0"
I20260619 06:58:02.560787 13401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 06:58:02.563230 13416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:02.564492 13401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:02.564699 13401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "fe605bc65bb44a58b6ed04de9790dfaa"
format_stamp: "Formatted at 2026-06-19 06:58:02 on dist-test-slave-10d0"
server_key: "46797b0964c9d265f91878ea49dd7635"
server_key_iv: "f559e1d0f5336d03daf1ccebca3a8c72"
server_key_version: "encryptionkey@0"
I20260619 06:58:02.564796 13401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:02.584340 13401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:02.587751 13401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:02.588094 13401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:02.595682 13401 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:39181
I20260619 06:58:02.595767 13478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:39181 every 8 connection(s)
I20260619 06:58:02.596702 13401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:02.599563 13479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:02.605216 13479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa: Bootstrap starting.
I20260619 06:58:02.606415 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 13401
I20260619 06:58:02.606503 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:02.606745 11689 external_mini_cluster.cc:1428] Setting key 6c5351234ee3f84fd33252c063f75c1f
I20260619 06:58:02.608006 13479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:02.608968 13479 log.cc:824] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:02.611251 13479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa: No bootstrap required, opened a new log
I20260619 06:58:02.615357 13479 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe605bc65bb44a58b6ed04de9790dfaa" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39181 } }
I20260619 06:58:02.615610 13479 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:02.615705 13479 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe605bc65bb44a58b6ed04de9790dfaa, State: Initialized, Role: FOLLOWER
Jun 19 06:58:02 dist-test-slave-10d0 krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:02.616253 13479 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe605bc65bb44a58b6ed04de9790dfaa" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39181 } }
I20260619 06:58:02.616420 13479 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:02.616490 13479 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:02.616608 13479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:02.617847 13479 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe605bc65bb44a58b6ed04de9790dfaa" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39181 } }
I20260619 06:58:02.618279 13479 leader_election.cc:302] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe605bc65bb44a58b6ed04de9790dfaa; no voters:
I20260619 06:58:02.618652 13479 leader_election.cc:288] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:02.618858 13484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:02.619131 13484 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [term 1 LEADER]: Becoming Leader. State: Replica: fe605bc65bb44a58b6ed04de9790dfaa, State: Running, Role: LEADER
I20260619 06:58:02.619436 13484 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe605bc65bb44a58b6ed04de9790dfaa" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39181 } }
I20260619 06:58:02.620118 13479 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:02.620743 13486 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe605bc65bb44a58b6ed04de9790dfaa. Latest consensus state: current_term: 1 leader_uuid: "fe605bc65bb44a58b6ed04de9790dfaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe605bc65bb44a58b6ed04de9790dfaa" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39181 } } }
I20260619 06:58:02.620963 13486 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:02.621209 13485 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe605bc65bb44a58b6ed04de9790dfaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe605bc65bb44a58b6ed04de9790dfaa" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39181 } } }
I20260619 06:58:02.621379 13485 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:02.621701 13493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:02.623385 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:02.607917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.11.106.126:39181)
0619 06:58:02.608420 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:02.608440 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:02.608483 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:58:02.609401 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:02.609427 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:02.609799 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:02.610166 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:02.610862 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.612307 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.613206 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.613924 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.614097 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:02.616779 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:02.616788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:02.616809 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:02.616847 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:02.619035 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:02.619566 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:02.619571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:02.619577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:02.619724 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:02.620096 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:02.620100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:02.620101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:02.620663 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:02.621778 (+ 1115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:02.621963 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 06:58:02.622360 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":166,"threads_started":1}
I20260619 06:58:02.625039 13493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:02.631482 13493 catalog_manager.cc:1347] Generated new cluster ID: 17e595f004d34a42a74d6c7d741938d4
I20260619 06:58:02.631561 13493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:02.652060 13493 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:02.653514 13493 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:02.669718 13493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa: Generated new TSK 0
I20260619 06:58:02.670615 13493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:58:02.705665 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39181
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:02.821004 13505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:02.821417 13505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:02.821525 13505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:02.825732 13505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:02.825796 13505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:02.825874 13505 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:58:02.825908 13505 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:58:02.825965 13505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:02.831458 13505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.106.126:39181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:02.832605 13505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:02.833681 13505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:02.844485 13510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:02.844485 13513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:02.844633 13511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:02.844883 13505 server_base.cc:1034] running on GCE node
I20260619 06:58:02.845268 13505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:02.845929 13505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:02.847110 13505 hybrid_clock.cc:648] HybridClock initialized: now 1781852282847080 us; error 47 us; skew 500 ppm
Jun 19 06:58:02 dist-test-slave-10d0 krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852282, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:02.849877 13505 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:58:02.850787 13505 webserver.cc:460] Webserver started at http://127.11.106.65:43615/ using document root <none> and password file <none>
I20260619 06:58:02.851361 13505 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:02.851455 13505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:02.851747 13505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:02.853931 13505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "04c882ed00b04dc4b6e939b3a9b4850c"
format_stamp: "Formatted at 2026-06-19 06:58:02 on dist-test-slave-10d0"
server_key: "9fdd0ee36d307f2fa340480ca352888e"
server_key_iv: "570971b1061c38d150c93177c997476a"
server_key_version: "encryptionkey@0"
I20260619 06:58:02.854511 13505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "04c882ed00b04dc4b6e939b3a9b4850c"
format_stamp: "Formatted at 2026-06-19 06:58:02 on dist-test-slave-10d0"
server_key: "9fdd0ee36d307f2fa340480ca352888e"
server_key_iv: "570971b1061c38d150c93177c997476a"
server_key_version: "encryptionkey@0"
I20260619 06:58:02.858574 13505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:02.861254 13520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:02.862577 13505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:02.862726 13505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "04c882ed00b04dc4b6e939b3a9b4850c"
format_stamp: "Formatted at 2026-06-19 06:58:02 on dist-test-slave-10d0"
server_key: "9fdd0ee36d307f2fa340480ca352888e"
server_key_iv: "570971b1061c38d150c93177c997476a"
server_key_version: "encryptionkey@0"
I20260619 06:58:02.862848 13505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:02.873986 13505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:02.876981 13505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:02.877184 13505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:02.878708 13505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:02.878782 13505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:02.878847 13505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:02.878898 13505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:58:02 dist-test-slave-10d0 krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852282, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:02.889657 13505 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:39945
I20260619 06:58:02.889706 13635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:39945 every 8 connection(s)
I20260619 06:58:02.890538 13505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:02.892419 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 13505
I20260619 06:58:02.892593 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:02.892880 11689 external_mini_cluster.cc:1428] Setting key b5f724c9471a5505896a62268978a2a4
I20260619 06:58:02.894044 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:02.879472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:54065 (local address 127.11.106.126:39181)
0619 06:58:02.879645 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:02.879650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:02.880983 (+ 1333us) server_negotiation.cc:407] Connection header received
0619 06:58:02.882181 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:02.882186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:02.882253 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:02.882356 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:02.884190 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.884948 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.886732 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.886976 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.887055 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:02.890039 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:02.890043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:02.890047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:02.890078 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:02.891718 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:02.892328 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:02.892333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:02.892334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:02.892423 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:02.892843 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:02.892847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:02.892849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:02.893091 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:02.893199 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:02.893740 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 06:58:02.893869 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:58:02.895044 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:02.880361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39181 (local address 127.11.106.65:54065)
0619 06:58:02.880826 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:02.880877 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:02.881944 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:02.882678 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:02.882690 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:02.883250 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:02.883995 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:02.884019 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.885057 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.885063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:02.886589 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:02.886600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.887138 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.887152 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:02.887383 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:02.888105 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:02.888126 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:02.889828 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:02.891835 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:02.891843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:02.891858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:02.892217 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:02.892518 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:02.892522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:02.892524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:02.892719 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:02.893195 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:02.893201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:02.893493 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 06:58:02.893686 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 06:58:02.894015 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":148,"threads_started":1}
I20260619 06:58:02.896528 13636 heartbeater.cc:347] Connected to a master server at 127.11.106.126:39181
I20260619 06:58:02.896713 13636 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:02.897353 13636 heartbeater.cc:511] Master 127.11.106.126:39181 requested a full tablet report, sending...
I20260619 06:58:02.899128 13433 ts_manager.cc:194] Registered new tserver with Master: 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945)
I20260619 06:58:02.900584 13433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:54065
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 06:58:02.907150 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:02.898731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39181 (local address 127.11.106.65:54335)
0619 06:58:02.899038 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:02.899053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:02.899154 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:02.899428 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:02.899432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:02.899649 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:02.899769 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:02.899777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.901043 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.901049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:02.902072 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:02.902086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.902716 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.902722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:02.902846 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:02.903349 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:02.903368 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:02.903829 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:02.905583 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:02.905590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:02.905595 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:02.905862 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:02.906135 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:02.906150 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:02.906154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:02.906301 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:02.906702 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:02.906705 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:02.906800 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 06:58:02.906888 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:58:02.906994 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":89,"threads_started":1}
I20260619 06:58:02.907375 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:02.898833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:54335 (local address 127.11.106.126:39181)
0619 06:58:02.898964 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:02.898970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:02.899135 (+ 165us) server_negotiation.cc:407] Connection header received
0619 06:58:02.899252 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:02.899259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:02.899319 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:02.899415 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:02.899904 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.900868 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.902221 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:02.902574 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:02.902682 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:02.903956 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:02.903959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:02.903962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:02.903982 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:02.905455 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:02.905953 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:02.905956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:02.905957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:02.906030 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:02.906417 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:02.906422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:02.906425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:02.906606 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:02.906682 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:02.906970 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 06:58:02.907115 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:58:02.923511 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39181
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:03.036244 13645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:03.036520 13645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:03.036643 13645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:03.040776 13645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:03.040832 13645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:03.040874 13645 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:58:03.040915 13645 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:58:03.040983 13645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:03.046360 13645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.106.126:39181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:03.047451 13645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:03.048478 13645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:03.058418 13651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:03.058432 13650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:03.058441 13653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:03.058897 13645 server_base.cc:1034] running on GCE node
I20260619 06:58:03.059281 13645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:03.059855 13645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:03.061009 13645 hybrid_clock.cc:648] HybridClock initialized: now 1781852283060974 us; error 45 us; skew 500 ppm
Jun 19 06:58:03 dist-test-slave-10d0 krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:03.063902 13645 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:58:03.064854 13645 webserver.cc:460] Webserver started at http://127.11.106.66:45209/ using document root <none> and password file <none>
I20260619 06:58:03.065424 13645 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:03.065485 13645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:03.065826 13645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:03.067998 13645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "fb5665dbe8a64d0987036226ea230695"
format_stamp: "Formatted at 2026-06-19 06:58:03 on dist-test-slave-10d0"
server_key: "8f25e81046af530c41f6adfb7f723c5e"
server_key_iv: "2d5227d4a09732c79cc9cb2410d4a0ae"
server_key_version: "encryptionkey@0"
I20260619 06:58:03.068686 13645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "fb5665dbe8a64d0987036226ea230695"
format_stamp: "Formatted at 2026-06-19 06:58:03 on dist-test-slave-10d0"
server_key: "8f25e81046af530c41f6adfb7f723c5e"
server_key_iv: "2d5227d4a09732c79cc9cb2410d4a0ae"
server_key_version: "encryptionkey@0"
I20260619 06:58:03.072293 13645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:03.074733 13660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:03.076025 13645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:03.076269 13645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "fb5665dbe8a64d0987036226ea230695"
format_stamp: "Formatted at 2026-06-19 06:58:03 on dist-test-slave-10d0"
server_key: "8f25e81046af530c41f6adfb7f723c5e"
server_key_iv: "2d5227d4a09732c79cc9cb2410d4a0ae"
server_key_version: "encryptionkey@0"
I20260619 06:58:03.076383 13645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:03.091270 13645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:03.094429 13645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:03.094626 13645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:03.096196 13645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:03.096283 13645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:03.096467 13645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:03.096591 13645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 06:58:03 dist-test-slave-10d0 krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:03.108130 13645 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:34653
I20260619 06:58:03.108207 13775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:34653 every 8 connection(s)
I20260619 06:58:03.109402 13645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:03.109766 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 13645
I20260619 06:58:03.109943 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:03.110267 11689 external_mini_cluster.cc:1428] Setting key a50fc23a6c8579266bdc87d155581674
I20260619 06:58:03.111588 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.097186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:36385 (local address 127.11.106.126:39181)
0619 06:58:03.097316 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.097325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.099073 (+ 1748us) server_negotiation.cc:407] Connection header received
0619 06:58:03.100358 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.100363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.100426 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.100520 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:03.102323 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.102931 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.104186 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.104435 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.104581 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.107273 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:03.107277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:03.107280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:03.107309 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:03.109155 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.109803 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.109809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.109814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.109910 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.110331 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.110335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.110337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.110559 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:03.110667 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.111337 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.111461 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:58:03.112416 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.098374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39181 (local address 127.11.106.66:36385)
0619 06:58:03.098906 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.098970 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.100122 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.100750 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.100759 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.101329 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:03.102114 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.102139 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.103068 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.103071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.104058 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.104067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.104614 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.104618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.104798 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.105444 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:03.105461 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:03.107085 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:03.109281 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.109286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.109298 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.109679 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.110011 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.110014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.110017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.110195 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.110862 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:03.110868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:03.111100 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.111265 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.111596 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":131,"threads_started":1}
I20260619 06:58:03.113898 13776 heartbeater.cc:347] Connected to a master server at 127.11.106.126:39181
I20260619 06:58:03.114086 13776 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:03.114741 13776 heartbeater.cc:511] Master 127.11.106.126:39181 requested a full tablet report, sending...
I20260619 06:58:03.115929 13431 ts_manager.cc:194] Registered new tserver with Master: fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653)
I20260619 06:58:03.116827 13431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:36385
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 06:58:03.124730 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.116086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39181 (local address 127.11.106.66:40529)
0619 06:58:03.116383 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.116398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.116487 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.116847 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.116850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.117118 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:03.117251 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.117259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.118434 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.118438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.119150 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.119160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.119813 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.119816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.119891 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.120596 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:03.120616 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:03.121119 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:03.122987 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.122990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.122992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.123278 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.123580 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.123583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.123584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.123663 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.124192 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:03.124195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:03.124273 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.124372 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.124485 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":125,"threads_started":1}
I20260619 06:58:03.124867 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.116459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:40529 (local address 127.11.106.126:39181)
0619 06:58:03.116652 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.116664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.116674 (+ 10us) server_negotiation.cc:407] Connection header received
0619 06:58:03.116705 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.116707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.116741 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.116830 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:03.117379 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.118280 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.119288 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.119673 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.119800 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.121267 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:03.121272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:03.121277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:03.121328 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:03.122886 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.123407 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.123410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.123411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.123489 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.123766 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.123769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.123770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.124092 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:03.124199 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.124466 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.124641 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:58:03.139501 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39181
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:03.255662 13785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:03.255934 13785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:03.255983 13785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:03.260150 13785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:03.260218 13785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:03.260260 13785 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:58:03.260303 13785 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:58:03.260365 13785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:03.265780 13785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.106.126:39181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:03.266919 13785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:03.268069 13785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:03.277907 13793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:03.277911 13791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:03.277907 13790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:03.278301 13785 server_base.cc:1034] running on GCE node
I20260619 06:58:03.278802 13785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:03.279503 13785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:03.280689 13785 hybrid_clock.cc:648] HybridClock initialized: now 1781852283280657 us; error 42 us; skew 500 ppm
Jun 19 06:58:03 dist-test-slave-10d0 krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:03.283414 13785 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:58:03.284359 13785 webserver.cc:460] Webserver started at http://127.11.106.67:40297/ using document root <none> and password file <none>
I20260619 06:58:03.285056 13785 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:03.285161 13785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:03.285432 13785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:03.287536 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "15453bc0fbc94f5e92bcf76c3720c0ce"
format_stamp: "Formatted at 2026-06-19 06:58:03 on dist-test-slave-10d0"
server_key: "07092ba6b97b47010d0c1a381cb0a793"
server_key_iv: "10dc5ef727e2afad344cbfb725d6e396"
server_key_version: "encryptionkey@0"
I20260619 06:58:03.288053 13785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "15453bc0fbc94f5e92bcf76c3720c0ce"
format_stamp: "Formatted at 2026-06-19 06:58:03 on dist-test-slave-10d0"
server_key: "07092ba6b97b47010d0c1a381cb0a793"
server_key_iv: "10dc5ef727e2afad344cbfb725d6e396"
server_key_version: "encryptionkey@0"
I20260619 06:58:03.292068 13785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:58:03.294730 13800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:03.296236 13785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:03.296361 13785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "15453bc0fbc94f5e92bcf76c3720c0ce"
format_stamp: "Formatted at 2026-06-19 06:58:03 on dist-test-slave-10d0"
server_key: "07092ba6b97b47010d0c1a381cb0a793"
server_key_iv: "10dc5ef727e2afad344cbfb725d6e396"
server_key_version: "encryptionkey@0"
I20260619 06:58:03.296475 13785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:03.321069 13785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:03.324695 13785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:03.324898 13785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:03.326483 13785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:03.326532 13785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:03.326572 13785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:03.326628 13785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:58:03 dist-test-slave-10d0 krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852283, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:03.337704 13915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:36219 every 8 connection(s)
I20260619 06:58:03.337646 13785 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:36219
I20260619 06:58:03.339156 13785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:03.341782 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.327409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:49533 (local address 127.11.106.126:39181)
0619 06:58:03.327542 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.327546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.328712 (+ 1166us) server_negotiation.cc:407] Connection header received
0619 06:58:03.329886 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.329890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.329951 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.330067 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:03.331757 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.332672 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.334581 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.334815 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.334904 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.338161 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:03.338165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:03.338194 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:03.338232 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:03.339880 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.340405 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.340408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.340409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.340483 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.340809 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.340812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.340813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.340989 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:03.341067 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.341562 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.341662 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:58:03.342545 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.328011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39181 (local address 127.11.106.67:49533)
0619 06:58:03.328501 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.328593 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.329620 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.330308 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.330318 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.330781 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:03.331545 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.331570 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.332822 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.332826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.334452 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.334463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.334985 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.334993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.335213 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.335953 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:03.335978 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:03.337954 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:03.340001 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.340025 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.340042 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.340330 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.340607 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.340609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.340610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.340730 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.341083 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:03.341088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:03.341335 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.341480 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.341720 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":125,"threads_started":1}
I20260619 06:58:03.343786 13916 heartbeater.cc:347] Connected to a master server at 127.11.106.126:39181
I20260619 06:58:03.343930 13916 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:03.344485 13916 heartbeater.cc:511] Master 127.11.106.126:39181 requested a full tablet report, sending...
I20260619 06:58:03.345583 13433 ts_manager.cc:194] Registered new tserver with Master: 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:03.346371 13433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:49533
I20260619 06:58:03.346551 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 13785
I20260619 06:58:03.346699 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:03.347025 11689 external_mini_cluster.cc:1428] Setting key 2d23018c93516d2b27263012369a8db9
I20260619 06:58:03.349119 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:58:03.353953 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.346089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:45049 (local address 127.11.106.126:39181)
0619 06:58:03.346217 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.346222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.346392 (+ 170us) server_negotiation.cc:407] Connection header received
0619 06:58:03.346558 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.346560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.346609 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.346689 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:03.347253 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.347975 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.349024 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.349331 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.349436 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.350502 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:03.350509 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:03.350515 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:03.350546 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:03.352053 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.352529 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.352532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.352533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.352659 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.353035 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.353038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.353040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.353227 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:03.353322 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.353715 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.353834 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 06:58:03.354035 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.346025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39181 (local address 127.11.106.67:45049)
0619 06:58:03.346290 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.346306 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.346411 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.346796 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.346807 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.347003 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:03.347120 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.347127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.348102 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.348106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.348898 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.348909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.349436 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.349439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.349513 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.350037 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:03.350048 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:03.350364 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:03.352160 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.352167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.352171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.352433 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.352769 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.352776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.352780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.352914 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.353334 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:03.353340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:03.353454 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.353576 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.353745 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
Jun 19 06:58:03 dist-test-slave-10d0 krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:58:03 dist-test-slave-10d0 krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:03.376683 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.368303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.11.106.126:39181)
0619 06:58:03.368437 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.368441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.368455 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:58:03.368497 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.368500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.368587 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.368697 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:03.369498 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.370048 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.370874 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.371082 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.371158 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.373597 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:03.373600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:03.373602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:03.373628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:03.374917 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.375294 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.375297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.375298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.375366 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.375657 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.375659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.375660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.375898 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:03.376003 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.376247 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.376346 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:58:03.380177 13433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:03.383616 13433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:03.398958 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.392595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:34653 (local address 127.0.0.1:53658)
0619 06:58:03.393540 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.393554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.393653 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.394609 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.394612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.394626 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:03.394735 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.394743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.395859 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.395861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.397408 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.397419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.398407 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.398410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.398515 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.398552 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.398688 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.398812 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":139,"threads_started":1}
I20260619 06:58:03.398970 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.393381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.11.106.66:34653)
0619 06:58:03.394049 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.394063 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.394088 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:58:03.394162 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.394166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.394310 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.394473 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:03.394859 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.395582 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.397538 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.398256 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.398408 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.398610 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.398710 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.398805 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":106,"threads_started":1}
I20260619 06:58:03.399148 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.392390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:39945 (local address 127.0.0.1:41336)
0619 06:58:03.392918 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.392979 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.393112 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.394494 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.394497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.394516 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:03.394640 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.394646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.396131 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.396136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.397722 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.397734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.398717 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.398721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.398839 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.398867 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.398919 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.398989 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":54,"thread_start_us":72,"threads_started":1}
I20260619 06:58:03.399691 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.393141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:36219 (local address 127.0.0.1:50068)
0619 06:58:03.393821 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.393834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.393906 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.394447 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.394453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.394477 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:03.394618 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.394625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.396049 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.396052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.397539 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.397547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.398905 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.398911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.398996 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.399015 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.399076 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.399127 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":114,"thread_start_us":96,"threads_started":1}
I20260619 06:58:03.399735 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.393393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.11.106.65:39945)
0619 06:58:03.394084 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.394098 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.394119 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:58:03.394201 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.394205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.394352 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.394814 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:03.394941 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.396020 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.397832 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.398588 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.399265 (+ 677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.399404 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.399454 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.399547 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":93,"threads_started":1}
I20260619 06:58:03.399931 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.393319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.11.106.67:36219)
0619 06:58:03.393856 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.393870 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.393898 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:58:03.394058 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.394064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.394280 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.394446 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:03.394743 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.395926 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.397707 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.398555 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.398678 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.398845 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.399221 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.399331 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":133,"threads_started":1}
I20260619 06:58:03.402688 13710 tablet_service.cc:1467] Processing CreateTablet for tablet aea62158a7ec4254bced6ce44a5963e8 (DEFAULT_TABLE table=test-table [id=9aab4a48d26d4c1a921b9602e1b6f217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:03.402685 13570 tablet_service.cc:1467] Processing CreateTablet for tablet aea62158a7ec4254bced6ce44a5963e8 (DEFAULT_TABLE table=test-table [id=9aab4a48d26d4c1a921b9602e1b6f217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:03.403862 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aea62158a7ec4254bced6ce44a5963e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:03.403863 13710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aea62158a7ec4254bced6ce44a5963e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:03.406064 13850 tablet_service.cc:1467] Processing CreateTablet for tablet aea62158a7ec4254bced6ce44a5963e8 (DEFAULT_TABLE table=test-table [id=9aab4a48d26d4c1a921b9602e1b6f217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:03.407084 13850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aea62158a7ec4254bced6ce44a5963e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:03.411484 13939 tablet_bootstrap.cc:492] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: Bootstrap starting.
I20260619 06:58:03.412806 13940 tablet_bootstrap.cc:492] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: Bootstrap starting.
I20260619 06:58:03.414270 13939 tablet_bootstrap.cc:654] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:03.414659 13940 tablet_bootstrap.cc:654] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:03.415499 13940 log.cc:824] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:03.415501 13939 log.cc:824] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:03.415839 13941 tablet_bootstrap.cc:492] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: Bootstrap starting.
I20260619 06:58:03.417598 13940 tablet_bootstrap.cc:492] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: No bootstrap required, opened a new log
I20260619 06:58:03.417791 13940 ts_tablet_manager.cc:1397] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20260619 06:58:03.417798 13939 tablet_bootstrap.cc:492] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: No bootstrap required, opened a new log
I20260619 06:58:03.417953 13941 tablet_bootstrap.cc:654] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:03.418058 13939 ts_tablet_manager.cc:1397] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 06:58:03.418769 13941 log.cc:824] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:03.420907 13941 tablet_bootstrap.cc:492] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: No bootstrap required, opened a new log
I20260619 06:58:03.421085 13941 ts_tablet_manager.cc:1397] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:58:03.421713 13940 raft_consensus.cc:348] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.421977 13940 raft_consensus.cc:374] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:03.422034 13940 raft_consensus.cc:729] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5665dbe8a64d0987036226ea230695, State: Initialized, Role: FOLLOWER
I20260619 06:58:03.422122 13939 raft_consensus.cc:348] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.422426 13939 raft_consensus.cc:374] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:03.422488 13939 raft_consensus.cc:729] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c882ed00b04dc4b6e939b3a9b4850c, State: Initialized, Role: FOLLOWER
I20260619 06:58:03.422533 13940 consensus_queue.cc:260] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.422986 13939 consensus_queue.cc:260] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.423579 13940 ts_tablet_manager.cc:1428] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:58:03.423687 13636 heartbeater.cc:503] Master 127.11.106.126:39181 was elected leader, sending a full tablet report...
I20260619 06:58:03.423934 13776 heartbeater.cc:503] Master 127.11.106.126:39181 was elected leader, sending a full tablet report...
I20260619 06:58:03.424041 13939 ts_tablet_manager.cc:1428] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260619 06:58:03.426143 13941 raft_consensus.cc:348] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.426401 13941 raft_consensus.cc:374] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:03.426460 13941 raft_consensus.cc:729] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15453bc0fbc94f5e92bcf76c3720c0ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:03.426898 13941 consensus_queue.cc:260] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.427510 13916 heartbeater.cc:503] Master 127.11.106.126:39181 was elected leader, sending a full tablet report...
I20260619 06:58:03.427842 13941 ts_tablet_manager.cc:1428] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:58:03.544596 13947 raft_consensus.cc:482] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:03.544610 13946 raft_consensus.cc:482] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:03.544829 13947 raft_consensus.cc:504] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.544883 13946 raft_consensus.cc:504] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.546001 13946 leader_election.cc:288] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:03.546326 13947 leader_election.cc:288] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945), fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653)
I20260619 06:58:03.551254 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.546920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:36527 (local address 127.11.106.66:34653)
0619 06:58:03.547203 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.547208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.547225 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:58:03.547276 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.547279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.547325 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.547398 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:03.547820 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.548424 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.549968 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.550552 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.550677 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.550771 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.551018 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.551101 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":190,"spinlock_wait_cycles":896}
I20260619 06:58:03.551401 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.546698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:45241 (local address 127.11.106.65:39945)
0619 06:58:03.546819 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.546824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.547073 (+ 249us) server_negotiation.cc:407] Connection header received
0619 06:58:03.547253 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.547256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.547294 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.547340 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:03.547713 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.548563 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.550072 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.550671 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.550753 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.550808 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.551197 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.551260 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:58:03.551568 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.546326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:47813 (local address 127.11.106.67:36219)
0619 06:58:03.546522 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.546528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.547073 (+ 545us) server_negotiation.cc:407] Connection header received
0619 06:58:03.547253 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.547256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.547294 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.547349 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:03.547713 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.548873 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.550254 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.550841 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.550976 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.551024 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.551366 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.551444 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":107}
I20260619 06:58:03.551659 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.546194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:34653 (local address 127.11.106.65:36527)
0619 06:58:03.546917 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.546931 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.547022 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.547526 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.547532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.547551 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:03.547691 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.547699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.548568 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.548571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.549829 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.549840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.550782 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.550787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.550885 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.550906 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.551404 (+ 498us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.551483 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":129,"threads_started":1}
I20260619 06:58:03.551816 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.546646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:39945 (local address 127.11.106.67:45241)
0619 06:58:03.546992 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.547005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.547143 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.547479 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.547482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.547496 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:03.547599 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.547605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.548701 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.548704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.549938 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.549948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.550899 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.550902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.551067 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.551089 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.551580 (+ 491us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.551651 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":36,"thread_start_us":86,"threads_started":1}
I20260619 06:58:03.552297 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.546267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:36219 (local address 127.11.106.65:47813)
0619 06:58:03.546991 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.547005 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.547140 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.547479 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.547482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.547496 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:03.547599 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.547605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.548995 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.548998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.550135 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.550143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.551104 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.551108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.551220 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.551239 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.552082 (+ 843us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.552154 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":98,"threads_started":1}
I20260619 06:58:03.552402 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.546201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:34653 (local address 127.11.106.67:38359)
0619 06:58:03.546569 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.546583 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.546686 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.547557 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.547561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.547578 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:03.547683 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.547689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.549301 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.549305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.550532 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.550542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.551985 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.551989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.552098 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.552119 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.552193 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.552262 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":91}
I20260619 06:58:03.552461 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea62158a7ec4254bced6ce44a5963e8" candidate_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" is_pre_election: true
I20260619 06:58:03.552578 13730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea62158a7ec4254bced6ce44a5963e8" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5665dbe8a64d0987036226ea230695" is_pre_election: true
I20260619 06:58:03.552740 13590 raft_consensus.cc:2454] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15453bc0fbc94f5e92bcf76c3720c0ce in term 0.
I20260619 06:58:03.552834 13730 raft_consensus.cc:2454] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 0.
I20260619 06:58:03.553036 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.546994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:38359 (local address 127.11.106.66:34653)
0619 06:58:03.547336 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.547345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.547358 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:58:03.547395 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.547397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.547437 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.547795 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:03.547896 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.549174 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.550668 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.551278 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.552005 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.552058 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.552733 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.552792 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":291,"spinlock_wait_cycles":768,"thread_start_us":130,"threads_started":1}
I20260619 06:58:03.553092 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea62158a7ec4254bced6ce44a5963e8" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" is_pre_election: true
I20260619 06:58:03.553231 13803 leader_election.cc:302] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, 15453bc0fbc94f5e92bcf76c3720c0ce; no voters:
I20260619 06:58:03.553272 13870 raft_consensus.cc:2454] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 0.
I20260619 06:58:03.553402 13730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea62158a7ec4254bced6ce44a5963e8" candidate_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5665dbe8a64d0987036226ea230695" is_pre_election: true
I20260619 06:58:03.553448 13524 leader_election.cc:302] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, fb5665dbe8a64d0987036226ea230695; no voters:
I20260619 06:58:03.553507 13730 raft_consensus.cc:2454] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15453bc0fbc94f5e92bcf76c3720c0ce in term 0.
I20260619 06:58:03.553551 13947 raft_consensus.cc:2781] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:03.553637 13947 raft_consensus.cc:482] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:03.553689 13947 raft_consensus.cc:3037] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:03.553719 13946 raft_consensus.cc:2781] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:03.553786 13946 raft_consensus.cc:482] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:03.553828 13946 raft_consensus.cc:3037] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:03.554996 13946 raft_consensus.cc:504] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.554996 13947 raft_consensus.cc:504] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.555361 13947 leader_election.cc:288] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [CANDIDATE]: Term 1 election: Requested vote from peers 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945), fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653)
I20260619 06:58:03.555361 13946 leader_election.cc:288] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 election: Requested vote from peers fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:03.555684 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea62158a7ec4254bced6ce44a5963e8" candidate_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c882ed00b04dc4b6e939b3a9b4850c"
I20260619 06:58:03.555771 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea62158a7ec4254bced6ce44a5963e8" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce"
I20260619 06:58:03.555898 13590 raft_consensus.cc:2379] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 15453bc0fbc94f5e92bcf76c3720c0ce in current term 1: Already voted for candidate 04c882ed00b04dc4b6e939b3a9b4850c in this term.
I20260619 06:58:03.555863 13729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea62158a7ec4254bced6ce44a5963e8" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5665dbe8a64d0987036226ea230695"
I20260619 06:58:03.555927 13870 raft_consensus.cc:2379] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 04c882ed00b04dc4b6e939b3a9b4850c in current term 1: Already voted for candidate 15453bc0fbc94f5e92bcf76c3720c0ce in this term.
I20260619 06:58:03.555920 13730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aea62158a7ec4254bced6ce44a5963e8" candidate_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5665dbe8a64d0987036226ea230695"
I20260619 06:58:03.555972 13729 raft_consensus.cc:3037] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:03.556488 13804 leader_election.cc:302] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15453bc0fbc94f5e92bcf76c3720c0ce; no voters: 04c882ed00b04dc4b6e939b3a9b4850c, fb5665dbe8a64d0987036226ea230695
I20260619 06:58:03.556690 13947 raft_consensus.cc:2726] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 06:58:03.557235 13729 raft_consensus.cc:2454] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 1.
I20260619 06:58:03.557520 13524 leader_election.cc:302] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, fb5665dbe8a64d0987036226ea230695; no voters: 15453bc0fbc94f5e92bcf76c3720c0ce
I20260619 06:58:03.557778 13946 raft_consensus.cc:2781] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:03.558022 13946 raft_consensus.cc:686] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 LEADER]: Becoming Leader. State: Replica: 04c882ed00b04dc4b6e939b3a9b4850c, State: Running, Role: LEADER
I20260619 06:58:03.558382 13946 consensus_queue.cc:237] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.561714 13433 catalog_manager.cc:5495] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c reported cstate change: term changed from 0 to 1, leader changed from <none> to 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65). New cstate: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } health_report { overall_health: UNKNOWN } } }
W20260619 06:58:03.590222 13917 tablet.cc:2369] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:58:03.610931 13777 tablet.cc:2369] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:58:03 dist-test-slave-10d0 krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852282, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:03.615175 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.606460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.11.106.126:39181)
0619 06:58:03.606613 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.606618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.606759 (+ 141us) server_negotiation.cc:407] Connection header received
0619 06:58:03.606927 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.606931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.606978 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.607066 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:03.607754 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.608285 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.609160 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.609379 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.609490 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.611963 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:03.611967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:03.611969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:03.611991 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:03.613377 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.613888 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.613891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.613892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.613968 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:03.614269 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:03.614274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:03.614281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:03.614458 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:03.614566 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.614864 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.615001 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:58:03.615211 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.606345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39181 (local address 127.0.0.1:41980)
0619 06:58:03.606655 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.606674 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.606819 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.607085 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.607088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.607525 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:03.607627 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.607634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.608412 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.608416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.609040 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.609049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.609491 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.609494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.609569 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.610124 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:03.610136 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:03.611830 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:03.613504 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.613513 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.613519 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.613790 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.614080 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:03.614085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:03.614087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:03.614173 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:03.614566 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:03.614575 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:03.614694 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.614832 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.615070 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":82,"thread_start_us":111,"threads_started":1}
I20260619 06:58:03.619503 13433 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:41980:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:58:03.620409 13433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:03.625494 13570 tablet_service.cc:1467] Processing CreateTablet for tablet ea550bcb78094d7d8e6ce30cf1a37e03 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4603a53dd232443fa96533ca31507121]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:58:03.625798 13850 tablet_service.cc:1467] Processing CreateTablet for tablet ea550bcb78094d7d8e6ce30cf1a37e03 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4603a53dd232443fa96533ca31507121]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:58:03.626426 13570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea550bcb78094d7d8e6ce30cf1a37e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:03.626544 13850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea550bcb78094d7d8e6ce30cf1a37e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:03.629307 13939 tablet_bootstrap.cc:492] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: Bootstrap starting.
I20260619 06:58:03.629443 13941 tablet_bootstrap.cc:492] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: Bootstrap starting.
I20260619 06:58:03.629644 13710 tablet_service.cc:1467] Processing CreateTablet for tablet ea550bcb78094d7d8e6ce30cf1a37e03 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4603a53dd232443fa96533ca31507121]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:58:03.630126 13710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea550bcb78094d7d8e6ce30cf1a37e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:03.630456 13941 tablet_bootstrap.cc:654] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:03.630509 13939 tablet_bootstrap.cc:654] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:03.631950 13939 tablet_bootstrap.cc:492] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: No bootstrap required, opened a new log
I20260619 06:58:03.632050 13939 ts_tablet_manager.cc:1397] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 06:58:03.632184 13941 tablet_bootstrap.cc:492] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: No bootstrap required, opened a new log
I20260619 06:58:03.632277 13941 ts_tablet_manager.cc:1397] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:03.632560 13939 raft_consensus.cc:348] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.632682 13939 raft_consensus.cc:374] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:03.632727 13939 raft_consensus.cc:729] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c882ed00b04dc4b6e939b3a9b4850c, State: Initialized, Role: FOLLOWER
I20260619 06:58:03.632725 13941 raft_consensus.cc:348] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.632831 13941 raft_consensus.cc:374] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:03.632874 13941 raft_consensus.cc:729] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15453bc0fbc94f5e92bcf76c3720c0ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:03.632890 13939 consensus_queue.cc:260] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.632954 13940 tablet_bootstrap.cc:492] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: Bootstrap starting.
I20260619 06:58:03.633008 13941 consensus_queue.cc:260] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.633163 13939 ts_tablet_manager.cc:1428] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 06:58:03.633287 13941 ts_tablet_manager.cc:1428] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:03.633255 13947 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:03.633412 13947 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:03.633396 13946 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:03.633503 13946 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:03.633921 13940 tablet_bootstrap.cc:654] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:03.635387 13940 tablet_bootstrap.cc:492] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: No bootstrap required, opened a new log
I20260619 06:58:03.635504 13940 ts_tablet_manager.cc:1397] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 06:58:03.636001 13940 raft_consensus.cc:348] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.636111 13940 raft_consensus.cc:374] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:03.636153 13940 raft_consensus.cc:729] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5665dbe8a64d0987036226ea230695, State: Initialized, Role: FOLLOWER
I20260619 06:58:03.636282 13940 consensus_queue.cc:260] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.636587 13940 ts_tablet_manager.cc:1428] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 06:58:03.636526 13945 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:03.636682 13945 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 06:58:03.642827 13637 tablet.cc:2369] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:03.976627 13946 raft_consensus.cc:482] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:03.976871 13946 raft_consensus.cc:504] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.977504 13946 leader_election.cc:288] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:03.978158 13729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea550bcb78094d7d8e6ce30cf1a37e03" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5665dbe8a64d0987036226ea230695" is_pre_election: true
I20260619 06:58:03.978379 13729 raft_consensus.cc:2454] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 0.
I20260619 06:58:03.978477 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea550bcb78094d7d8e6ce30cf1a37e03" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" is_pre_election: true
I20260619 06:58:03.978683 13870 raft_consensus.cc:2454] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 0.
I20260619 06:58:03.978734 13524 leader_election.cc:302] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, fb5665dbe8a64d0987036226ea230695; no voters:
I20260619 06:58:03.978930 13946 raft_consensus.cc:2781] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:03.979012 13946 raft_consensus.cc:482] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:03.979068 13946 raft_consensus.cc:3037] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:03.980072 13946 raft_consensus.cc:504] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.980463 13946 leader_election.cc:288] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 election: Requested vote from peers fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:03.980846 13729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea550bcb78094d7d8e6ce30cf1a37e03" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5665dbe8a64d0987036226ea230695"
I20260619 06:58:03.980934 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea550bcb78094d7d8e6ce30cf1a37e03" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce"
I20260619 06:58:03.981001 13729 raft_consensus.cc:3037] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:03.981093 13870 raft_consensus.cc:3037] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:03.981958 13729 raft_consensus.cc:2454] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 1.
I20260619 06:58:03.982105 13870 raft_consensus.cc:2454] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 1.
I20260619 06:58:03.982267 13524 leader_election.cc:302] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, fb5665dbe8a64d0987036226ea230695; no voters:
I20260619 06:58:03.982462 13946 raft_consensus.cc:2781] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:03.982590 13946 raft_consensus.cc:686] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 LEADER]: Becoming Leader. State: Replica: 04c882ed00b04dc4b6e939b3a9b4850c, State: Running, Role: LEADER
I20260619 06:58:03.982733 13946 consensus_queue.cc:237] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:03.983510 13952 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: TxnStatusTablet state changed. Reason: New leader 04c882ed00b04dc4b6e939b3a9b4850c. Latest consensus state: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:03.983623 13952 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:58:03.983939 13964 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:58:03.984201 13964 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:58:03.984730 13433 catalog_manager.cc:5495] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c reported cstate change: term changed from 0 to 1, leader changed from <none> to 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65). New cstate: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:03.998049 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.994196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.11.106.65:39945)
0619 06:58:03.994325 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:03.994329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:03.994344 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:03.994511 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:03.994514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:03.994580 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:03.994682 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:03.994919 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.995422 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.996401 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.996738 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.996851 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.997226 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:03.997662 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:03.997782 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:03.997857 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 06:58:03.997906 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 06:58:03.998049 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:03.994065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:39945 (local address 127.0.0.1:41342)
0619 06:58:03.994233 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:03.994252 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:03.994358 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:03.994696 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:03.994699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:03.994707 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:03.994815 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.994820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.995537 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.995542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.996287 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:03.996296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:03.996851 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:03.996855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:03.996947 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:03.997044 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:03.997754 (+ 710us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:03.997770 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 06:58:03.997842 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 06:58:03.997933 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:58:04.003343 13870 raft_consensus.cc:1261] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Refusing update from remote peer 04c882ed00b04dc4b6e939b3a9b4850c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:04.003374 13729 raft_consensus.cc:1261] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Refusing update from remote peer 04c882ed00b04dc4b6e939b3a9b4850c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:04.003973 13952 consensus_queue.cc:1035] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:04.004169 13946 consensus_queue.cc:1035] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c [LEADER]: Connected to new peer: Peer: permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:04.007743 13945 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: TxnStatusTablet state changed. Reason: New leader 04c882ed00b04dc4b6e939b3a9b4850c. Latest consensus state: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:04.007879 13945 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:04.008119 13947 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: TxnStatusTablet state changed. Reason: New leader 04c882ed00b04dc4b6e939b3a9b4850c. Latest consensus state: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:04.008244 13947 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:04.011159 13952 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:04.011283 13952 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:58:04.013296 13946 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:04.013411 13946 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P 04c882ed00b04dc4b6e939b3a9b4850c: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:58:04.014350 13945 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:04.014467 13945 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P fb5665dbe8a64d0987036226ea230695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:04.014412 13947 tablet_replica.cc:439] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } } }
I20260619 06:58:04.014513 13947 tablet_replica.cc:442] T ea550bcb78094d7d8e6ce30cf1a37e03 P 15453bc0fbc94f5e92bcf76c3720c0ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:04.014633 13969 mvcc.cc:205] Tried to move back new op lower bound from 7298466955271110656 to 7298466955195019264. Current Snapshot: MvccSnapshot[applied={T|T < 7298466955271110656}]
I20260619 06:58:04.022449 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.018181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.11.106.65:39945)
0619 06:58:04.018490 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:04.018494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:04.018510 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:04.018698 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:04.018702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:04.018751 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:04.018797 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:04.019325 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.019957 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.021339 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.021598 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.021725 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.022010 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:04.022087 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:04.022168 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:04.022261 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 06:58:04.022318 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260619 06:58:04.033078 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.029837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:56879 (local address 127.11.106.65:39945)
0619 06:58:04.029951 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:04.029955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:04.030154 (+ 199us) server_negotiation.cc:407] Connection header received
0619 06:58:04.030302 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:04.030305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:04.030357 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:04.030444 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:04.030694 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.031182 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.032001 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.032210 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.032314 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.032621 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:04.032688 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:04.032769 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:04.032872 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 06:58:04.032909 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":44}
I20260619 06:58:04.033090 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.029738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:39945 (local address 127.11.106.65:56879)
0619 06:58:04.030072 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:04.030083 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:04.030208 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:04.030451 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:04.030454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:04.030460 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:04.030595 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.030601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.031281 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.031287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.031889 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.031896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.032321 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.032324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.032413 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.032450 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:04.032771 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:04.032780 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:58:04.032850 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:58:04.032922 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":18,"thread_start_us":94,"threads_started":1}
I20260619 06:58:04.039778 13729 raft_consensus.cc:1261] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Refusing update from remote peer 04c882ed00b04dc4b6e939b3a9b4850c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:04.039799 13870 raft_consensus.cc:1261] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Refusing update from remote peer 04c882ed00b04dc4b6e939b3a9b4850c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:04.040190 13952 consensus_queue.cc:1035] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [LEADER]: Connected to new peer: Peer: permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:04.040339 13946 consensus_queue.cc:1035] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:04.045305 13968 mvcc.cc:205] Tried to move back new op lower bound from 7298466955423502336 to 7298466953458274304. Current Snapshot: MvccSnapshot[applied={T|T < 7298466955423502336}]
I20260619 06:58:04.045760 13967 mvcc.cc:205] Tried to move back new op lower bound from 7298466955423502336 to 7298466953458274304. Current Snapshot: MvccSnapshot[applied={T|T < 7298466955423502336}]
I20260619 06:58:04.085114 13432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:04.085307 13432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:04.088112 13432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa: Sending DeleteTablet for 3 replicas of tablet aea62158a7ec4254bced6ce44a5963e8
I20260619 06:58:04.088869 13569 tablet_service.cc:1514] Processing DeleteTablet for tablet aea62158a7ec4254bced6ce44a5963e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:04 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:41336
I20260619 06:58:04.088884 13710 tablet_service.cc:1514] Processing DeleteTablet for tablet aea62158a7ec4254bced6ce44a5963e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:04 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:53658
I20260619 06:58:04.089159 13850 tablet_service.cc:1514] Processing DeleteTablet for tablet aea62158a7ec4254bced6ce44a5963e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:04 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50068
I20260619 06:58:04.089318 13984 tablet_replica.cc:330] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: stopping tablet replica
I20260619 06:58:04.089624 13984 raft_consensus.cc:2229] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:04.089865 13985 tablet_replica.cc:330] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: stopping tablet replica
I20260619 06:58:04.089968 13984 raft_consensus.cc:2258] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:04.090193 13985 raft_consensus.cc:2229] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:04.090126 13432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:58:04.090394 13985 raft_consensus.cc:2258] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 06:58:04.090564 13432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:04.090750 13986 tablet_replica.cc:330] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: stopping tablet replica
I20260619 06:58:04.091032 13986 raft_consensus.cc:2229] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:04.091259 13986 raft_consensus.cc:2258] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:04.091431 13984 ts_tablet_manager.cc:1905] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:04.093323 13985 ts_tablet_manager.cc:1905] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:04.094528 13984 ts_tablet_manager.cc:1918] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:04.094619 13984 log.cc:1196] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/aea62158a7ec4254bced6ce44a5963e8
I20260619 06:58:04.095069 13984 ts_tablet_manager.cc:1939] T aea62158a7ec4254bced6ce44a5963e8 P 04c882ed00b04dc4b6e939b3a9b4850c: Deleting consensus metadata
I20260619 06:58:04.096318 13985 ts_tablet_manager.cc:1918] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:04.096424 13985 log.cc:1196] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/aea62158a7ec4254bced6ce44a5963e8
I20260619 06:58:04.096766 13985 ts_tablet_manager.cc:1939] T aea62158a7ec4254bced6ce44a5963e8 P fb5665dbe8a64d0987036226ea230695: Deleting consensus metadata
I20260619 06:58:04.096768 13710 tablet_service.cc:1467] Processing CreateTablet for tablet 576dedf03898439298b48d4381b30838 (DEFAULT_TABLE table=test-table [id=0db97c543ee54002b94f654c85259887]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:04.096900 13419 catalog_manager.cc:4841] TS 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945): tablet aea62158a7ec4254bced6ce44a5963e8 (table test-table [id=9aab4a48d26d4c1a921b9602e1b6f217]) successfully deleted
I20260619 06:58:04.097052 13710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 576dedf03898439298b48d4381b30838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:04.097425 13569 tablet_service.cc:1467] Processing CreateTablet for tablet 576dedf03898439298b48d4381b30838 (DEFAULT_TABLE table=test-table [id=0db97c543ee54002b94f654c85259887]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:04.097679 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 576dedf03898439298b48d4381b30838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:04.097987 13420 catalog_manager.cc:4841] TS fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653): tablet aea62158a7ec4254bced6ce44a5963e8 (table test-table [id=9aab4a48d26d4c1a921b9602e1b6f217]) successfully deleted
I20260619 06:58:04.099022 13940 tablet_bootstrap.cc:492] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: Bootstrap starting.
I20260619 06:58:04.099843 13940 tablet_bootstrap.cc:654] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:04.099910 13939 tablet_bootstrap.cc:492] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: Bootstrap starting.
I20260619 06:58:04.100780 13939 tablet_bootstrap.cc:654] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:04.101109 13850 tablet_service.cc:1467] Processing CreateTablet for tablet 576dedf03898439298b48d4381b30838 (DEFAULT_TABLE table=test-table [id=0db97c543ee54002b94f654c85259887]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:04.101377 13850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 576dedf03898439298b48d4381b30838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:04.101938 13940 tablet_bootstrap.cc:492] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: No bootstrap required, opened a new log
I20260619 06:58:04.102043 13940 ts_tablet_manager.cc:1397] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 06:58:04.102450 13940 raft_consensus.cc:348] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:04.102571 13940 raft_consensus.cc:374] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:04.102612 13940 raft_consensus.cc:729] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5665dbe8a64d0987036226ea230695, State: Initialized, Role: FOLLOWER
I20260619 06:58:04.102739 13940 consensus_queue.cc:260] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:04.102996 13940 ts_tablet_manager.cc:1428] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:04.103799 13941 tablet_bootstrap.cc:492] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: Bootstrap starting.
I20260619 06:58:04.104686 13941 tablet_bootstrap.cc:654] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:04.105224 13939 tablet_bootstrap.cc:492] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: No bootstrap required, opened a new log
I20260619 06:58:04.105316 13939 ts_tablet_manager.cc:1397] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 06:58:04.105793 13939 raft_consensus.cc:348] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:04.105914 13939 raft_consensus.cc:374] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:04.105952 13939 raft_consensus.cc:729] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c882ed00b04dc4b6e939b3a9b4850c, State: Initialized, Role: FOLLOWER
I20260619 06:58:04.106076 13939 consensus_queue.cc:260] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:04.106202 13941 tablet_bootstrap.cc:492] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: No bootstrap required, opened a new log
I20260619 06:58:04.106272 13941 ts_tablet_manager.cc:1397] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 06:58:04.106335 13939 ts_tablet_manager.cc:1428] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:04.106704 13941 raft_consensus.cc:348] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:04.106802 13941 raft_consensus.cc:374] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:04.106832 13941 raft_consensus.cc:729] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15453bc0fbc94f5e92bcf76c3720c0ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:04.106950 13941 consensus_queue.cc:260] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:04.107235 13941 ts_tablet_manager.cc:1428] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:58:04.110378 13986 ts_tablet_manager.cc:1905] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:04.113272 13986 ts_tablet_manager.cc:1918] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:04.113369 13986 log.cc:1196] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/aea62158a7ec4254bced6ce44a5963e8
I20260619 06:58:04.113675 13986 ts_tablet_manager.cc:1939] T aea62158a7ec4254bced6ce44a5963e8 P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting consensus metadata
I20260619 06:58:04.114830 13419 catalog_manager.cc:4841] TS 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219): tablet aea62158a7ec4254bced6ce44a5963e8 (table test-table [id=9aab4a48d26d4c1a921b9602e1b6f217]) successfully deleted
I20260619 06:58:04.230835 13945 raft_consensus.cc:482] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:04.231009 13945 raft_consensus.cc:504] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:04.232359 13945 leader_election.cc:288] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:04.237121 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.232674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:39945 (local address 127.11.106.66:56637)
0619 06:58:04.232913 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:04.232930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:04.233071 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:04.233888 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:04.233890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:04.233906 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:04.234036 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.234043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.234836 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.234838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.235619 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.235627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.236677 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.236681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.236781 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.236806 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:58:04.236878 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 06:58:04.236951 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260619 06:58:04.237612 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.233378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:56637 (local address 127.11.106.65:39945)
0619 06:58:04.233502 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:04.233508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:04.233523 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:04.233600 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:04.233606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:04.233670 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:04.233756 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:04.234180 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.234712 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.235755 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.236378 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.236503 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.236602 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:04.237406 (+ 804us) server_negotiation.cc:299] Negotiation successful
0619 06:58:04.237491 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 06:58:04.238018 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576dedf03898439298b48d4381b30838" candidate_uuid: "fb5665dbe8a64d0987036226ea230695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" is_pre_election: true
I20260619 06:58:04.238191 13590 raft_consensus.cc:2454] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb5665dbe8a64d0987036226ea230695 in term 0.
I20260619 06:58:04.238632 13663 leader_election.cc:302] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, fb5665dbe8a64d0987036226ea230695; no voters:
I20260619 06:58:04.238943 13945 raft_consensus.cc:2781] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:04.239002 13945 raft_consensus.cc:482] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:04.239044 13945 raft_consensus.cc:3037] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:04.240286 13945 raft_consensus.cc:504] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:04.240631 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.234066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:38031 (local address 127.11.106.67:36219)
0619 06:58:04.235859 (+ 1793us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:04.235874 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:04.235886 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:58:04.235930 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:04.235933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:04.236015 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:04.236119 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:04.236406 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.237315 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.238956 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.239673 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.239806 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.239865 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:04.240344 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 06:58:04.240444 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1712,"thread_start_us":91,"threads_started":1}
I20260619 06:58:04.240904 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.233029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:36219 (local address 127.11.106.66:38031)
0619 06:58:04.233560 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:04.233643 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:04.233717 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:04.236128 (+ 2411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:04.236132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:04.236151 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:04.236251 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.236260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.237472 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.237475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.238799 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.238811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.239933 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.239938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.240041 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.240065 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:58:04.240137 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 06:58:04.240229 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":53,"threads_started":1}
I20260619 06:58:04.241053 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576dedf03898439298b48d4381b30838" candidate_uuid: "fb5665dbe8a64d0987036226ea230695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c882ed00b04dc4b6e939b3a9b4850c"
I20260619 06:58:04.241186 13590 raft_consensus.cc:3037] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:04.241353 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576dedf03898439298b48d4381b30838" candidate_uuid: "fb5665dbe8a64d0987036226ea230695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" is_pre_election: true
I20260619 06:58:04.241514 13870 raft_consensus.cc:2454] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb5665dbe8a64d0987036226ea230695 in term 0.
I20260619 06:58:04.242334 13590 raft_consensus.cc:2454] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb5665dbe8a64d0987036226ea230695 in term 1.
I20260619 06:58:04.242638 13663 leader_election.cc:302] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, fb5665dbe8a64d0987036226ea230695; no voters:
I20260619 06:58:04.242684 13945 leader_election.cc:288] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [CANDIDATE]: Term 1 election: Requested vote from peers 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:04.242969 13945 raft_consensus.cc:2781] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:04.243196 13945 raft_consensus.cc:686] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 1 LEADER]: Becoming Leader. State: Replica: fb5665dbe8a64d0987036226ea230695, State: Running, Role: LEADER
I20260619 06:58:04.243203 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576dedf03898439298b48d4381b30838" candidate_uuid: "fb5665dbe8a64d0987036226ea230695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce"
I20260619 06:58:04.243343 13870 raft_consensus.cc:3037] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:04.243458 13945 consensus_queue.cc:237] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:04.244562 13870 raft_consensus.cc:2454] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb5665dbe8a64d0987036226ea230695 in term 1.
I20260619 06:58:04.246367 13433 catalog_manager.cc:5495] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb5665dbe8a64d0987036226ea230695 (127.11.106.66). New cstate: current_term: 1 leader_uuid: "fb5665dbe8a64d0987036226ea230695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:04.279592 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.274714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.11.106.66:34653)
0619 06:58:04.275006 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:04.275014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:04.275087 (+ 73us) server_negotiation.cc:407] Connection header received
0619 06:58:04.275303 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:04.275306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:04.275366 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:04.275823 (+ 457us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:04.275936 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.276898 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.277756 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.277994 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.278310 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.278612 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:04.279093 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:04.279276 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:04.279331 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 06:58:04.279389 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":127,"threads_started":1}
I20260619 06:58:04.291045 13998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.287320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:39945 (local address 127.11.106.66:51631)
0619 06:58:04.287630 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:04.287642 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:04.287734 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:04.288003 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:04.288005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:04.288011 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:04.288113 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.288117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.289093 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.289096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.289712 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.289720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.290298 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.290301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.290377 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.290423 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:04.290760 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:04.290766 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:04.290822 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 06:58:04.290888 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":113,"threads_started":1}
I20260619 06:58:04.291050 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.287430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:51631 (local address 127.11.106.65:39945)
0619 06:58:04.287667 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:04.287673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:04.287692 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:58:04.287845 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:04.287848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:04.287905 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:04.287990 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:04.288213 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.288987 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.289838 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.290167 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.290278 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.290566 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:04.290665 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:04.290745 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:04.290860 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 06:58:04.290914 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 06:58:04.298046 13870 raft_consensus.cc:1261] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Refusing update from remote peer fb5665dbe8a64d0987036226ea230695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:04.298110 13590 raft_consensus.cc:1261] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Refusing update from remote peer fb5665dbe8a64d0987036226ea230695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:04.298620 13993 consensus_queue.cc:1035] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:04.298856 13945 consensus_queue.cc:1035] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:04.320783 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.316477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:34653 (local address 127.11.106.65:40981)
0619 06:58:04.316627 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:04.316643 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:04.316729 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:04.317281 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:04.317284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:04.317292 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:04.317388 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.317395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.318484 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.318491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.319269 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:04.319280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.319932 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.319936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:04.320024 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.320056 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:04.320405 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:04.320412 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:58:04.320468 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 06:58:04.320591 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 06:58:04.321141 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:04.316772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:40981 (local address 127.11.106.66:34653)
0619 06:58:04.316915 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:04.316922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:04.316941 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:58:04.316998 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:04.317001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:04.317062 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:04.317150 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:04.317515 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.318338 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.319388 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:04.319755 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:04.319852 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:04.320170 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:04.320250 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:04.320341 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:04.320961 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 06:58:04.321016 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 06:58:04.321866 13969 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:58:05.884285 13431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:05.884508 13431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:05.886934 13431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa: Sending DeleteTablet for 3 replicas of tablet 576dedf03898439298b48d4381b30838
I20260619 06:58:05.887640 13569 tablet_service.cc:1514] Processing DeleteTablet for tablet 576dedf03898439298b48d4381b30838 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:05 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:41336
I20260619 06:58:05.887792 13710 tablet_service.cc:1514] Processing DeleteTablet for tablet 576dedf03898439298b48d4381b30838 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:05 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:53658
I20260619 06:58:05.888036 13850 tablet_service.cc:1514] Processing DeleteTablet for tablet 576dedf03898439298b48d4381b30838 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:05 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50068
I20260619 06:58:05.889029 14026 tablet_replica.cc:330] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: stopping tablet replica
I20260619 06:58:05.889184 14026 raft_consensus.cc:2229] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:05.889578 14026 raft_consensus.cc:2258] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:05.890301 14027 tablet_replica.cc:330] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: stopping tablet replica
I20260619 06:58:05.890414 14027 raft_consensus.cc:2229] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:05.890512 14027 raft_consensus.cc:2258] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:05.892374 14027 ts_tablet_manager.cc:1905] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:05.893337 14026 ts_tablet_manager.cc:1905] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:05.894970 14027 ts_tablet_manager.cc:1918] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:58:05.895226 14027 log.cc:1196] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/576dedf03898439298b48d4381b30838
I20260619 06:58:05.895594 14027 ts_tablet_manager.cc:1939] T 576dedf03898439298b48d4381b30838 P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting consensus metadata
I20260619 06:58:05.896037 14025 tablet_replica.cc:330] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: stopping tablet replica
I20260619 06:58:05.896210 14025 raft_consensus.cc:2229] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:05.896356 14025 raft_consensus.cc:2258] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:05.896441 13419 catalog_manager.cc:4841] TS 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219): tablet 576dedf03898439298b48d4381b30838 (table test-table [id=0db97c543ee54002b94f654c85259887]) successfully deleted
I20260619 06:58:05.897213 14025 ts_tablet_manager.cc:1905] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:05.897271 14026 ts_tablet_manager.cc:1918] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:58:05.897360 14026 log.cc:1196] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/576dedf03898439298b48d4381b30838
I20260619 06:58:05.897742 14026 ts_tablet_manager.cc:1939] T 576dedf03898439298b48d4381b30838 P fb5665dbe8a64d0987036226ea230695: Deleting consensus metadata
I20260619 06:58:05.898512 13420 catalog_manager.cc:4841] TS fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653): tablet 576dedf03898439298b48d4381b30838 (table test-table [id=0db97c543ee54002b94f654c85259887]) successfully deleted
I20260619 06:58:05.899722 14025 ts_tablet_manager.cc:1918] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:58:05.899788 14025 log.cc:1196] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/576dedf03898439298b48d4381b30838
I20260619 06:58:05.900032 14025 ts_tablet_manager.cc:1939] T 576dedf03898439298b48d4381b30838 P 04c882ed00b04dc4b6e939b3a9b4850c: Deleting consensus metadata
I20260619 06:58:05.900704 13419 catalog_manager.cc:4841] TS 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945): tablet 576dedf03898439298b48d4381b30838 (table test-table [id=0db97c543ee54002b94f654c85259887]) successfully deleted
Jun 19 06:58:05 dist-test-slave-10d0 krb5kdc[13388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:58:05 dist-test-slave-10d0 krb5kdc[13388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852285, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:05.926436 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:05.911202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.11.106.126:39181)
0619 06:58:05.911544 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:05.911551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:05.911568 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:58:05.911766 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:05.911769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:05.911816 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:05.911899 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:05.912834 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:05.913434 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:05.914429 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:05.914657 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:05.914745 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:05.922806 (+ 8061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:05.922810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:05.922812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:05.922846 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:05.924450 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:05.924992 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:05.924996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:05.924997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:05.925101 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:05.925405 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:05.925408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:05.925409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:05.925617 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:05.925773 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:05.926084 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 06:58:05.926209 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":149,"threads_started":1}
I20260619 06:58:05.929188 13431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:05.929575 13431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:05.934206 13569 tablet_service.cc:1467] Processing CreateTablet for tablet de1605af9a0a4b80bf91c00b213fc1fd (DEFAULT_TABLE table=test-table [id=67c30fd6017f41cb9d2ef7dce37c90d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:05.934223 13710 tablet_service.cc:1467] Processing CreateTablet for tablet de1605af9a0a4b80bf91c00b213fc1fd (DEFAULT_TABLE table=test-table [id=67c30fd6017f41cb9d2ef7dce37c90d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:05.934527 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1605af9a0a4b80bf91c00b213fc1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:05.934531 13710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1605af9a0a4b80bf91c00b213fc1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:05.934478 13850 tablet_service.cc:1467] Processing CreateTablet for tablet de1605af9a0a4b80bf91c00b213fc1fd (DEFAULT_TABLE table=test-table [id=67c30fd6017f41cb9d2ef7dce37c90d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:05.934721 13850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1605af9a0a4b80bf91c00b213fc1fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:05.936715 14036 tablet_bootstrap.cc:492] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: Bootstrap starting.
I20260619 06:58:05.937539 14037 tablet_bootstrap.cc:492] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: Bootstrap starting.
I20260619 06:58:05.937603 14038 tablet_bootstrap.cc:492] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: Bootstrap starting.
I20260619 06:58:05.937724 14036 tablet_bootstrap.cc:654] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:05.938482 14037 tablet_bootstrap.cc:654] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:05.938584 14038 tablet_bootstrap.cc:654] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:05.939056 14036 tablet_bootstrap.cc:492] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: No bootstrap required, opened a new log
I20260619 06:58:05.939214 14036 ts_tablet_manager.cc:1397] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:05.939606 14037 tablet_bootstrap.cc:492] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: No bootstrap required, opened a new log
I20260619 06:58:05.939715 14037 ts_tablet_manager.cc:1397] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:58:05.939693 14036 raft_consensus.cc:348] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:05.939843 14036 raft_consensus.cc:374] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:05.939889 14036 raft_consensus.cc:729] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c882ed00b04dc4b6e939b3a9b4850c, State: Initialized, Role: FOLLOWER
I20260619 06:58:05.939940 14038 tablet_bootstrap.cc:492] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: No bootstrap required, opened a new log
I20260619 06:58:05.940027 14038 ts_tablet_manager.cc:1397] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:05.940021 14036 consensus_queue.cc:260] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:05.940145 14037 raft_consensus.cc:348] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:05.940240 14037 raft_consensus.cc:374] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:05.940272 14037 raft_consensus.cc:729] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5665dbe8a64d0987036226ea230695, State: Initialized, Role: FOLLOWER
I20260619 06:58:05.940328 14036 ts_tablet_manager.cc:1428] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:05.940389 14037 consensus_queue.cc:260] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:05.940498 14038 raft_consensus.cc:348] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:05.940624 14038 raft_consensus.cc:374] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:05.940666 14038 raft_consensus.cc:729] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15453bc0fbc94f5e92bcf76c3720c0ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:05.940744 14037 ts_tablet_manager.cc:1428] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:05.940784 14038 consensus_queue.cc:260] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:05.941166 14038 ts_tablet_manager.cc:1428] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:06.280788 14016 raft_consensus.cc:482] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:06.280938 14016 raft_consensus.cc:504] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.281435 14016 leader_election.cc:288] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:06.281953 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1605af9a0a4b80bf91c00b213fc1fd" candidate_uuid: "fb5665dbe8a64d0987036226ea230695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" is_pre_election: true
I20260619 06:58:06.282028 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1605af9a0a4b80bf91c00b213fc1fd" candidate_uuid: "fb5665dbe8a64d0987036226ea230695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" is_pre_election: true
I20260619 06:58:06.282119 13590 raft_consensus.cc:2454] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb5665dbe8a64d0987036226ea230695 in term 0.
I20260619 06:58:06.282207 13869 raft_consensus.cc:2454] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb5665dbe8a64d0987036226ea230695 in term 0.
I20260619 06:58:06.282517 13663 leader_election.cc:302] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, fb5665dbe8a64d0987036226ea230695; no voters:
I20260619 06:58:06.282728 14016 raft_consensus.cc:2781] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:06.282802 14016 raft_consensus.cc:482] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:06.282832 14016 raft_consensus.cc:3037] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:06.283874 14016 raft_consensus.cc:504] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.284286 14016 leader_election.cc:288] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [CANDIDATE]: Term 1 election: Requested vote from peers 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:06.284749 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1605af9a0a4b80bf91c00b213fc1fd" candidate_uuid: "fb5665dbe8a64d0987036226ea230695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce"
I20260619 06:58:06.284745 13590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1605af9a0a4b80bf91c00b213fc1fd" candidate_uuid: "fb5665dbe8a64d0987036226ea230695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c882ed00b04dc4b6e939b3a9b4850c"
I20260619 06:58:06.284885 13869 raft_consensus.cc:3037] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:06.284899 13590 raft_consensus.cc:3037] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:06.286083 13869 raft_consensus.cc:2454] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb5665dbe8a64d0987036226ea230695 in term 1.
I20260619 06:58:06.286083 13590 raft_consensus.cc:2454] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb5665dbe8a64d0987036226ea230695 in term 1.
I20260619 06:58:06.286378 13663 leader_election.cc:302] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15453bc0fbc94f5e92bcf76c3720c0ce, fb5665dbe8a64d0987036226ea230695; no voters:
I20260619 06:58:06.286548 14016 raft_consensus.cc:2781] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:06.286638 14016 raft_consensus.cc:686] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 1 LEADER]: Becoming Leader. State: Replica: fb5665dbe8a64d0987036226ea230695, State: Running, Role: LEADER
I20260619 06:58:06.286793 14016 consensus_queue.cc:237] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.288532 13431 catalog_manager.cc:5495] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb5665dbe8a64d0987036226ea230695 (127.11.106.66). New cstate: current_term: 1 leader_uuid: "fb5665dbe8a64d0987036226ea230695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:06.320147 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:06.317099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.11.106.66:34653)
0619 06:58:06.317403 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:06.317411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:06.317423 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:58:06.317463 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:06.317465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:06.317504 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:06.317584 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:06.317793 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:06.318299 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:06.319149 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:06.319368 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:06.319442 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:06.319743 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:06.319813 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:06.319869 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:06.319986 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 06:58:06.320029 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":161,"threads_started":1}
I20260619 06:58:06.327173 13869 raft_consensus.cc:1261] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Refusing update from remote peer fb5665dbe8a64d0987036226ea230695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:06.327173 13590 raft_consensus.cc:1261] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Refusing update from remote peer fb5665dbe8a64d0987036226ea230695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:06.327606 14003 consensus_queue.cc:1035] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:06.327811 14016 consensus_queue.cc:1035] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:06.358546 13429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:06.358690 13429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:06.360724 13429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa: Sending DeleteTablet for 3 replicas of tablet de1605af9a0a4b80bf91c00b213fc1fd
I20260619 06:58:06.361343 13710 tablet_service.cc:1514] Processing DeleteTablet for tablet de1605af9a0a4b80bf91c00b213fc1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:06 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:53658
I20260619 06:58:06.361364 13850 tablet_service.cc:1514] Processing DeleteTablet for tablet de1605af9a0a4b80bf91c00b213fc1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:06 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50068
I20260619 06:58:06.361531 14027 tablet_replica.cc:330] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: stopping tablet replica
I20260619 06:58:06.361533 14026 tablet_replica.cc:330] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: stopping tablet replica
I20260619 06:58:06.361640 14026 raft_consensus.cc:2229] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:06.361637 14027 raft_consensus.cc:2229] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:06.361742 14027 raft_consensus.cc:2258] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:06.361874 14026 raft_consensus.cc:2258] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:06.362138 13569 tablet_service.cc:1514] Processing DeleteTablet for tablet de1605af9a0a4b80bf91c00b213fc1fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:06 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:41336
I20260619 06:58:06.362098 13429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:58:06.362336 14025 tablet_replica.cc:330] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: stopping tablet replica
I20260619 06:58:06.362447 14025 raft_consensus.cc:2229] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:06.362447 14027 ts_tablet_manager.cc:1905] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 06:58:06.362504 13429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:06.362545 14025 raft_consensus.cc:2258] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:06.362569 14026 ts_tablet_manager.cc:1905] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:06.363448 14025 ts_tablet_manager.cc:1905] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:06.365149 14026 ts_tablet_manager.cc:1918] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:06.365167 14027 ts_tablet_manager.cc:1918] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:06.365217 14026 log.cc:1196] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/de1605af9a0a4b80bf91c00b213fc1fd
I20260619 06:58:06.365236 14027 log.cc:1196] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/de1605af9a0a4b80bf91c00b213fc1fd
I20260619 06:58:06.365482 14026 ts_tablet_manager.cc:1939] T de1605af9a0a4b80bf91c00b213fc1fd P fb5665dbe8a64d0987036226ea230695: Deleting consensus metadata
I20260619 06:58:06.365532 14027 ts_tablet_manager.cc:1939] T de1605af9a0a4b80bf91c00b213fc1fd P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting consensus metadata
I20260619 06:58:06.366259 13420 catalog_manager.cc:4841] TS fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653): tablet de1605af9a0a4b80bf91c00b213fc1fd (table test-table [id=67c30fd6017f41cb9d2ef7dce37c90d6]) successfully deleted
I20260619 06:58:06.366597 14025 ts_tablet_manager.cc:1918] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:06.366621 13419 catalog_manager.cc:4841] TS 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219): tablet de1605af9a0a4b80bf91c00b213fc1fd (table test-table [id=67c30fd6017f41cb9d2ef7dce37c90d6]) successfully deleted
I20260619 06:58:06.366683 14025 log.cc:1196] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/de1605af9a0a4b80bf91c00b213fc1fd
I20260619 06:58:06.366922 14025 ts_tablet_manager.cc:1939] T de1605af9a0a4b80bf91c00b213fc1fd P 04c882ed00b04dc4b6e939b3a9b4850c: Deleting consensus metadata
I20260619 06:58:06.367671 13710 tablet_service.cc:1467] Processing CreateTablet for tablet 984ed494b519434da9f708949dc885d2 (DEFAULT_TABLE table=test-table [id=589025d5e94540e7852379f3ac5b9d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:06.367671 13569 tablet_service.cc:1467] Processing CreateTablet for tablet 984ed494b519434da9f708949dc885d2 (DEFAULT_TABLE table=test-table [id=589025d5e94540e7852379f3ac5b9d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:06.367919 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 984ed494b519434da9f708949dc885d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:06.367919 13710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 984ed494b519434da9f708949dc885d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:06.368803 13850 tablet_service.cc:1467] Processing CreateTablet for tablet 984ed494b519434da9f708949dc885d2 (DEFAULT_TABLE table=test-table [id=589025d5e94540e7852379f3ac5b9d11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:06.369113 13850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 984ed494b519434da9f708949dc885d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:06.370170 14037 tablet_bootstrap.cc:492] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: Bootstrap starting.
I20260619 06:58:06.371147 14038 tablet_bootstrap.cc:492] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: Bootstrap starting.
I20260619 06:58:06.371148 14037 tablet_bootstrap.cc:654] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:06.371745 14036 tablet_bootstrap.cc:492] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c: Bootstrap starting.
I20260619 06:58:06.372037 14038 tablet_bootstrap.cc:654] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:06.372363 14037 tablet_bootstrap.cc:492] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: No bootstrap required, opened a new log
I20260619 06:58:06.372465 14037 ts_tablet_manager.cc:1397] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:06.372696 14036 tablet_bootstrap.cc:654] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:06.372701 13419 catalog_manager.cc:4841] TS 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945): tablet de1605af9a0a4b80bf91c00b213fc1fd (table test-table [id=67c30fd6017f41cb9d2ef7dce37c90d6]) successfully deleted
I20260619 06:58:06.372954 14037 raft_consensus.cc:348] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.373064 14037 raft_consensus.cc:374] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:06.373106 14037 raft_consensus.cc:729] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5665dbe8a64d0987036226ea230695, State: Initialized, Role: FOLLOWER
I20260619 06:58:06.373170 14038 tablet_bootstrap.cc:492] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: No bootstrap required, opened a new log
I20260619 06:58:06.373260 14038 ts_tablet_manager.cc:1397] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:58:06.373234 14037 consensus_queue.cc:260] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.373517 14037 ts_tablet_manager.cc:1428] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:58:06.373731 14038 raft_consensus.cc:348] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.373850 14038 raft_consensus.cc:374] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:06.373888 14038 raft_consensus.cc:729] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15453bc0fbc94f5e92bcf76c3720c0ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:06.373931 14036 tablet_bootstrap.cc:492] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c: No bootstrap required, opened a new log
I20260619 06:58:06.374008 14036 ts_tablet_manager.cc:1397] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 06:58:06.374018 14038 consensus_queue.cc:260] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.374289 14038 ts_tablet_manager.cc:1428] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:06.374408 14036 raft_consensus.cc:348] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.374514 14036 raft_consensus.cc:374] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:06.374547 14036 raft_consensus.cc:729] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c882ed00b04dc4b6e939b3a9b4850c, State: Initialized, Role: FOLLOWER
I20260619 06:58:06.374663 14036 consensus_queue.cc:260] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.374908 14036 ts_tablet_manager.cc:1428] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:58:06.466328 13952 raft_consensus.cc:482] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:06.466514 13952 raft_consensus.cc:504] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.467041 13952 leader_election.cc:288] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:06.467572 13729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "984ed494b519434da9f708949dc885d2" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5665dbe8a64d0987036226ea230695" is_pre_election: true
I20260619 06:58:06.467572 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "984ed494b519434da9f708949dc885d2" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" is_pre_election: true
I20260619 06:58:06.467736 13869 raft_consensus.cc:2454] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 0.
I20260619 06:58:06.467767 13729 raft_consensus.cc:2454] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 0.
I20260619 06:58:06.468111 13523 leader_election.cc:302] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, 15453bc0fbc94f5e92bcf76c3720c0ce; no voters:
I20260619 06:58:06.468327 13952 raft_consensus.cc:2781] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:06.468458 13952 raft_consensus.cc:482] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:06.468513 13952 raft_consensus.cc:3037] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:06.469854 13952 raft_consensus.cc:504] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.470207 13952 leader_election.cc:288] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 election: Requested vote from peers fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653), 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219)
I20260619 06:58:06.470602 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "984ed494b519434da9f708949dc885d2" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce"
I20260619 06:58:06.470625 13729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "984ed494b519434da9f708949dc885d2" candidate_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5665dbe8a64d0987036226ea230695"
I20260619 06:58:06.470718 13869 raft_consensus.cc:3037] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:06.470817 13729 raft_consensus.cc:3037] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:06.471879 13869 raft_consensus.cc:2454] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 1.
I20260619 06:58:06.471913 13729 raft_consensus.cc:2454] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c882ed00b04dc4b6e939b3a9b4850c in term 1.
I20260619 06:58:06.472152 13523 leader_election.cc:302] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c882ed00b04dc4b6e939b3a9b4850c, 15453bc0fbc94f5e92bcf76c3720c0ce; no voters:
I20260619 06:58:06.472355 13952 raft_consensus.cc:2781] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:06.472457 13952 raft_consensus.cc:686] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 LEADER]: Becoming Leader. State: Replica: 04c882ed00b04dc4b6e939b3a9b4850c, State: Running, Role: LEADER
I20260619 06:58:06.472653 13952 consensus_queue.cc:237] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } }
I20260619 06:58:06.474267 13429 catalog_manager.cc:5495] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c reported cstate change: term changed from 0 to 1, leader changed from <none> to 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65). New cstate: current_term: 1 leader_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c882ed00b04dc4b6e939b3a9b4850c" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 39945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:06.509209 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:06.505794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.11.106.65:39945)
0619 06:58:06.506134 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:06.506141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:06.506154 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:58:06.506206 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:06.506208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:06.506255 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:06.506331 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:06.506556 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:06.507083 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:06.508063 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:06.508305 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:06.508400 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:06.508672 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:06.508746 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:06.508806 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:06.508957 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 06:58:06.509005 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":154,"threads_started":1}
I20260619 06:58:06.515942 13869 raft_consensus.cc:1261] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Refusing update from remote peer 04c882ed00b04dc4b6e939b3a9b4850c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:06.516142 13729 raft_consensus.cc:1261] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Refusing update from remote peer 04c882ed00b04dc4b6e939b3a9b4850c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:06.516414 13952 consensus_queue.cc:1035] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [LEADER]: Connected to new peer: Peer: permanent_uuid: "15453bc0fbc94f5e92bcf76c3720c0ce" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:06.516608 14006 consensus_queue.cc:1035] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb5665dbe8a64d0987036226ea230695" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:58:06.530143 14004 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:58:08.081167 13431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:08.081312 13431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:08.083597 13431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe605bc65bb44a58b6ed04de9790dfaa: Sending DeleteTablet for 3 replicas of tablet 984ed494b519434da9f708949dc885d2
I20260619 06:58:08.084141 13569 tablet_service.cc:1514] Processing DeleteTablet for tablet 984ed494b519434da9f708949dc885d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:08 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:41336
I20260619 06:58:08.084151 13710 tablet_service.cc:1514] Processing DeleteTablet for tablet 984ed494b519434da9f708949dc885d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:08 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:53658
I20260619 06:58:08.084183 13850 tablet_service.cc:1514] Processing DeleteTablet for tablet 984ed494b519434da9f708949dc885d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:08 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50068
I20260619 06:58:08.084522 14078 tablet_replica.cc:330] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: stopping tablet replica
I20260619 06:58:08.084702 14080 tablet_replica.cc:330] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c: stopping tablet replica
I20260619 06:58:08.084722 14078 raft_consensus.cc:2229] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:08.084863 14080 raft_consensus.cc:2229] T 984ed494b519434da9f708949dc885d2 P 04c882ed00b04dc4b6e939b3a9b4850c [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:08.084906 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 13505
I20260619 06:58:08.085634 14079 tablet_replica.cc:330] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: stopping tablet replica
I20260619 06:58:08.085793 14079 raft_consensus.cc:2229] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:08.085929 14079 raft_consensus.cc:2258] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:08.086581 14078 raft_consensus.cc:2258] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:08.086669 14079 ts_tablet_manager.cc:1905] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:08.089000 14078 ts_tablet_manager.cc:1905] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:08.090052 14079 ts_tablet_manager.cc:1918] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:58:08.090139 14079 log.cc:1196] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/984ed494b519434da9f708949dc885d2
I20260619 06:58:08.090420 14079 ts_tablet_manager.cc:1939] T 984ed494b519434da9f708949dc885d2 P 15453bc0fbc94f5e92bcf76c3720c0ce: Deleting consensus metadata
I20260619 06:58:08.091248 14078 ts_tablet_manager.cc:1918] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:58:08.091243 13419 catalog_manager.cc:4841] TS 15453bc0fbc94f5e92bcf76c3720c0ce (127.11.106.67:36219): tablet 984ed494b519434da9f708949dc885d2 (table test-table [id=589025d5e94540e7852379f3ac5b9d11]) successfully deleted
I20260619 06:58:08.091338 14078 log.cc:1196] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/984ed494b519434da9f708949dc885d2
I20260619 06:58:08.091569 14078 ts_tablet_manager.cc:1939] T 984ed494b519434da9f708949dc885d2 P fb5665dbe8a64d0987036226ea230695: Deleting consensus metadata
I20260619 06:58:08.092206 13420 catalog_manager.cc:4841] TS fb5665dbe8a64d0987036226ea230695 (127.11.106.66:34653): tablet 984ed494b519434da9f708949dc885d2 (table test-table [id=589025d5e94540e7852379f3ac5b9d11]) successfully deleted
W20260619 06:58:08.094552 13419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:39945 (error 108)
I20260619 06:58:08.094805 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 13645
I20260619 06:58:08.095953 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:08.095573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:39945 (local address 127.0.0.1:41362)
0619 06:58:08.095798 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:08.095876 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
W20260619 06:58:08.096175 13419 catalog_manager.cc:4584] TS 04c882ed00b04dc4b6e939b3a9b4850c (127.11.106.65:39945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 984ed494b519434da9f708949dc885d2: Network error: Client connection negotiation failed: client connection to 127.11.106.65:39945: connect: Connection refused (error 111)
I20260619 06:58:08.102368 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 13785
I20260619 06:58:08.108704 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 13401
2026-06-19T06:58:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6876 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:08 dist-test-slave-10d0 krb5kdc[14085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:08 dist-test-slave-10d0 krb5kdc[14085](info): set up 2 sockets
Jun 19 06:58:08 dist-test-slave-10d0 krb5kdc[14085](info): commencing operation
krb5kdc: starting...
W20260619 06:58:09.169051 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:09 dist-test-slave-10d0 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:09.291767 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46613
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:46613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:09.404948 14098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:09.405274 14098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:09.405355 14098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:09.409448 14098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:09.409530 14098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:09.409569 14098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:09.409627 14098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:09.409663 14098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:09.415076 14098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:46613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:09.416177 14098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:09.417274 14098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:09.426029 14103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:09.426023 14104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:09.426024 14106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:09.426172 14098 server_base.cc:1034] running on GCE node
I20260619 06:58:09.426858 14098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:09.428078 14098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:09.429268 14098 hybrid_clock.cc:648] HybridClock initialized: now 1781852289429251 us; error 28 us; skew 500 ppm
Jun 19 06:58:09 dist-test-slave-10d0 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:09.432082 14098 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:09.433077 14098 webserver.cc:460] Webserver started at http://127.11.106.126:42539/ using document root <none> and password file <none>
I20260619 06:58:09.433674 14098 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:09.433766 14098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:09.434000 14098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:09.436169 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "daba0e1b91584b5f9ee4764313e7ace3"
format_stamp: "Formatted at 2026-06-19 06:58:09 on dist-test-slave-10d0"
server_key: "2de143ced768f6521ae83832c2fded51"
server_key_iv: "cb2f9bfcafd637b430efd755aad1e31d"
server_key_version: "encryptionkey@0"
I20260619 06:58:09.436803 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "daba0e1b91584b5f9ee4764313e7ace3"
format_stamp: "Formatted at 2026-06-19 06:58:09 on dist-test-slave-10d0"
server_key: "2de143ced768f6521ae83832c2fded51"
server_key_iv: "cb2f9bfcafd637b430efd755aad1e31d"
server_key_version: "encryptionkey@0"
I20260619 06:58:09.440594 14098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:58:09.442960 14113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:09.444141 14098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:09.444283 14098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "daba0e1b91584b5f9ee4764313e7ace3"
format_stamp: "Formatted at 2026-06-19 06:58:09 on dist-test-slave-10d0"
server_key: "2de143ced768f6521ae83832c2fded51"
server_key_iv: "cb2f9bfcafd637b430efd755aad1e31d"
server_key_version: "encryptionkey@0"
I20260619 06:58:09.444409 14098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:09.462821 14098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:09.465857 14098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:09.466068 14098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:09.473448 14098 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:46613
I20260619 06:58:09.473459 14165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:46613 every 8 connection(s)
I20260619 06:58:09.474422 14098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:09.477334 14166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:09.478395 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 14098
I20260619 06:58:09.478503 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:09.478814 11689 external_mini_cluster.cc:1428] Setting key 07cb69e4fd42dc7830c21218e8d7c77b
I20260619 06:58:09.484195 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3: Bootstrap starting.
I20260619 06:58:09.486893 14166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:09.487658 14166 log.cc:824] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:58:09 dist-test-slave-10d0 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:09.489943 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3: No bootstrap required, opened a new log
I20260619 06:58:09.493671 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:09.480346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.11.106.126:46613)
0619 06:58:09.480805 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:09.480824 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:09.480869 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:58:09.481703 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:09.481726 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:09.482145 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:09.482529 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:09.483221 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:09.484663 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:09.485512 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:09.486186 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:09.486345 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:09.489028 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:09.489033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:09.489047 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:09.489073 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:09.490729 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:09.491212 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:09.491217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:09.491221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:09.491344 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:09.491621 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:09.491623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:09.491624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:09.492002 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:09.492150 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:09.492527 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 06:58:09.492918 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":155,"threads_started":1}
I20260619 06:58:09.494132 14166 raft_consensus.cc:348] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba0e1b91584b5f9ee4764313e7ace3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46613 } }
I20260619 06:58:09.494379 14166 raft_consensus.cc:374] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:09.494448 14166 raft_consensus.cc:729] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daba0e1b91584b5f9ee4764313e7ace3, State: Initialized, Role: FOLLOWER
I20260619 06:58:09.494891 14166 consensus_queue.cc:260] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba0e1b91584b5f9ee4764313e7ace3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46613 } }
I20260619 06:58:09.495048 14166 raft_consensus.cc:388] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:09.495132 14166 raft_consensus.cc:482] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:09.495257 14166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:09.496167 14166 raft_consensus.cc:504] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba0e1b91584b5f9ee4764313e7ace3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46613 } }
I20260619 06:58:09.496572 14166 leader_election.cc:302] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daba0e1b91584b5f9ee4764313e7ace3; no voters:
I20260619 06:58:09.496902 14166 leader_election.cc:288] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:09.497071 14171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:09.497397 14171 raft_consensus.cc:686] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [term 1 LEADER]: Becoming Leader. State: Replica: daba0e1b91584b5f9ee4764313e7ace3, State: Running, Role: LEADER
I20260619 06:58:09.497756 14171 consensus_queue.cc:237] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba0e1b91584b5f9ee4764313e7ace3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46613 } }
I20260619 06:58:09.498196 14166 sys_catalog.cc:564] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:09.499440 14173 sys_catalog.cc:455] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daba0e1b91584b5f9ee4764313e7ace3. Latest consensus state: current_term: 1 leader_uuid: "daba0e1b91584b5f9ee4764313e7ace3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba0e1b91584b5f9ee4764313e7ace3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46613 } } }
I20260619 06:58:09.499567 14173 sys_catalog.cc:458] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:09.499993 14172 sys_catalog.cc:455] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daba0e1b91584b5f9ee4764313e7ace3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daba0e1b91584b5f9ee4764313e7ace3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46613 } } }
I20260619 06:58:09.500135 14172 sys_catalog.cc:458] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3 [sys.catalog]: This master's current role is: LEADER
W20260619 06:58:09.503449 14186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:58:09.503535 14186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:58:09.503607 14181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:09.504328 14181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:09.510419 14181 catalog_manager.cc:1347] Generated new cluster ID: af8a58aa0e004ea59bed048045e628fb
I20260619 06:58:09.510527 14181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:09.525952 14181 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:09.526932 14181 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:09.533824 14181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daba0e1b91584b5f9ee4764313e7ace3: Generated new TSK 0
I20260619 06:58:09.534552 14181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:58:09.572391 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46613
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:09.708024 14192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:09.708328 14192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:09.708375 14192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:09.712643 14192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:09.712718 14192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:09.712787 14192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:09.718292 14192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:09.719445 14192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:09.720562 14192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:09.730729 14200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:09.730837 14197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:09.730738 14198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:09.730857 14192 server_base.cc:1034] running on GCE node
I20260619 06:58:09.731470 14192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:09.732126 14192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:09.733309 14192 hybrid_clock.cc:648] HybridClock initialized: now 1781852289733292 us; error 35 us; skew 500 ppm
Jun 19 06:58:09 dist-test-slave-10d0 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:09.736156 14192 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:58:09.737167 14192 webserver.cc:460] Webserver started at http://127.11.106.65:44251/ using document root <none> and password file <none>
I20260619 06:58:09.737768 14192 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:09.737869 14192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:09.738111 14192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:09.740234 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "b8e8966fa25b4eb6b9a30bdf567309c0"
format_stamp: "Formatted at 2026-06-19 06:58:09 on dist-test-slave-10d0"
server_key: "aff479b1ced2ab2bd8e108dcfb2d7cef"
server_key_iv: "080418451fa58f11452d9de6a6fd0047"
server_key_version: "encryptionkey@0"
I20260619 06:58:09.740860 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "b8e8966fa25b4eb6b9a30bdf567309c0"
format_stamp: "Formatted at 2026-06-19 06:58:09 on dist-test-slave-10d0"
server_key: "aff479b1ced2ab2bd8e108dcfb2d7cef"
server_key_iv: "080418451fa58f11452d9de6a6fd0047"
server_key_version: "encryptionkey@0"
I20260619 06:58:09.744851 14192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:58:09.747617 14207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:09.749157 14192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:09.749315 14192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "b8e8966fa25b4eb6b9a30bdf567309c0"
format_stamp: "Formatted at 2026-06-19 06:58:09 on dist-test-slave-10d0"
server_key: "aff479b1ced2ab2bd8e108dcfb2d7cef"
server_key_iv: "080418451fa58f11452d9de6a6fd0047"
server_key_version: "encryptionkey@0"
I20260619 06:58:09.749442 14192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:09.766072 14192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:09.769575 14192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:09.769809 14192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:09.770479 14192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:09.771387 14192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:09.771461 14192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:09.771534 14192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:09.771586 14192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:09.781515 14192 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:36705
I20260619 06:58:09.781531 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:36705 every 8 connection(s)
I20260619 06:58:09.782562 14192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:09.790616 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 14192
I20260619 06:58:09.790773 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:09.791049 11689 external_mini_cluster.cc:1428] Setting key 85de539be4f88101f2cb22f6d10756c5
Jun 19 06:58:09 dist-test-slave-10d0 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:09.797837 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:09.784607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:39649 (local address 127.11.106.126:46613)
0619 06:58:09.784741 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:09.784746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:09.785675 (+ 929us) server_negotiation.cc:407] Connection header received
0619 06:58:09.787017 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:09.787021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:09.787080 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:09.787168 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:09.788460 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:09.789092 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:09.790271 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:09.790526 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:09.790647 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:09.793626 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:09.793629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:09.793632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:09.793655 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:09.795403 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:09.795933 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:09.795940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:09.795944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:09.796037 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:09.796427 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:09.796433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:09.796437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:09.796733 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:09.796850 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:09.797483 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 06:58:09.797627 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:58:09.798745 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:09.785022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46613 (local address 127.11.106.65:39649)
0619 06:58:09.785510 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:09.785557 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:09.786785 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:09.787376 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:09.787383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:09.787757 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:09.788282 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:09.788296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:09.789229 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:09.789232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:09.790134 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:09.790143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:09.790704 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:09.790709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:09.790938 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:09.791630 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:09.791654 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:09.793452 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:09.795527 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:09.795532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:09.795546 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:09.795821 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:09.796164 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:09.796167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:09.796169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:09.796318 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:09.796870 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:09.796876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:09.797125 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 06:58:09.797389 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 06:58:09.797798 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":106,"threads_started":1}
I20260619 06:58:09.800453 14321 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46613
I20260619 06:58:09.800745 14321 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:09.801272 14321 heartbeater.cc:511] Master 127.11.106.126:46613 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 06:58:09.803115 14126 ts_manager.cc:194] Registered new tserver with Master: b8e8966fa25b4eb6b9a30bdf567309c0 (127.11.106.65:36705)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 06:58:09.804852 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:39649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:58:09.819635 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46613
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:09.933101 14326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:09.933411 14326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:09.933506 14326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:09.937819 14326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:09.937903 14326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:09.937991 14326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:09.943612 14326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:09.944882 14326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:09.945992 14326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:09.956395 14332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:09.956395 14331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:09.956437 14326 server_base.cc:1034] running on GCE node
W20260619 06:58:09.956401 14334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:09.957132 14326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:09.957835 14326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:09.959049 14326 hybrid_clock.cc:648] HybridClock initialized: now 1781852289959017 us; error 44 us; skew 500 ppm
Jun 19 06:58:09 dist-test-slave-10d0 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:09.962034 14326 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:58:09.962985 14326 webserver.cc:460] Webserver started at http://127.11.106.66:45403/ using document root <none> and password file <none>
I20260619 06:58:09.963590 14326 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:09.963683 14326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:09.963933 14326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:09.966158 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "5bfafcd09c084b9a8954589ae4a06219"
format_stamp: "Formatted at 2026-06-19 06:58:09 on dist-test-slave-10d0"
server_key: "a39e3f2f01d750f8055826be5cb4aeed"
server_key_iv: "a8b43baf290436f141e6d91d907c941f"
server_key_version: "encryptionkey@0"
I20260619 06:58:09.966720 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "5bfafcd09c084b9a8954589ae4a06219"
format_stamp: "Formatted at 2026-06-19 06:58:09 on dist-test-slave-10d0"
server_key: "a39e3f2f01d750f8055826be5cb4aeed"
server_key_iv: "a8b43baf290436f141e6d91d907c941f"
server_key_version: "encryptionkey@0"
I20260619 06:58:09.970553 14326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:09.973106 14341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:09.974372 14326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:09.974527 14326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "5bfafcd09c084b9a8954589ae4a06219"
format_stamp: "Formatted at 2026-06-19 06:58:09 on dist-test-slave-10d0"
server_key: "a39e3f2f01d750f8055826be5cb4aeed"
server_key_iv: "a8b43baf290436f141e6d91d907c941f"
server_key_version: "encryptionkey@0"
I20260619 06:58:09.974643 14326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:09.984603 14326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:09.987484 14326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:09.987699 14326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:09.988305 14326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:09.989598 14326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:09.989678 14326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:09.989746 14326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:09.989777 14326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:09.999009 14326 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:39219
I20260619 06:58:09.999068 14454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:39219 every 8 connection(s)
I20260619 06:58:09.999989 14326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:10.006177 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 14326
I20260619 06:58:10.006266 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:10.006531 11689 external_mini_cluster.cc:1428] Setting key 89b415052bfd7ad22f720c94769e84c7
Jun 19 06:58:10 dist-test-slave-10d0 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852289, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:10.015440 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:10.002063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:40667 (local address 127.11.106.126:46613)
0619 06:58:10.002222 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:10.002227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:10.003158 (+ 931us) server_negotiation.cc:407] Connection header received
0619 06:58:10.004693 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:10.004696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:10.004749 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:10.004832 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:10.006097 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:10.006752 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:10.007938 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:10.008187 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:10.008310 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:10.011360 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:10.011363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:10.011366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:10.011391 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:10.013040 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:10.013740 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:10.013742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:10.013743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:10.013844 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:10.014277 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:10.014279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:10.014280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:10.014456 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:10.014528 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:10.015202 (+ 674us) server_negotiation.cc:299] Negotiation successful
0619 06:58:10.015312 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:58:10.016816 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:10.002509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46613 (local address 127.11.106.66:40667)
0619 06:58:10.002985 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:10.003031 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:10.004386 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:10.005035 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:10.005043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:10.005418 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:10.005921 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:10.005935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:10.006886 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:10.006889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:10.007803 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:10.007812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:10.008350 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:10.008355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:10.008531 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:10.009227 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:10.009251 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:10.011167 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:10.013171 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:10.013180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:10.013203 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:10.013631 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:10.013942 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:10.013949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:10.013953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:10.014166 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:10.014564 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:10.014573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:10.014926 (+ 353us) client_negotiation.cc:769] Sending connection context
0619 06:58:10.015171 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 06:58:10.015577 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 06:58:10.018092 14455 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46613
I20260619 06:58:10.018326 14455 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:10.018802 14455 heartbeater.cc:511] Master 127.11.106.126:46613 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 06:58:10.020004 14126 ts_manager.cc:194] Registered new tserver with Master: 5bfafcd09c084b9a8954589ae4a06219 (127.11.106.66:39219)
I20260619 06:58:10.020677 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:40667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:58:10.036875 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46613
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:10.150255 14460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:10.150543 14460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:10.150620 14460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:10.155088 14460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:10.155161 14460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:10.155258 14460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:10.160743 14460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:10.161926 14460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:10.163013 14460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:10.172761 14465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:10.172842 14468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:10.172780 14466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:10.173012 14460 server_base.cc:1034] running on GCE node
I20260619 06:58:10.173374 14460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:10.174070 14460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:10.175252 14460 hybrid_clock.cc:648] HybridClock initialized: now 1781852290175218 us; error 55 us; skew 500 ppm
Jun 19 06:58:10 dist-test-slave-10d0 krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852290, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:10.178043 14460 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:58:10.178982 14460 webserver.cc:460] Webserver started at http://127.11.106.67:36221/ using document root <none> and password file <none>
I20260619 06:58:10.179533 14460 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:10.179621 14460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:10.179852 14460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:10.181954 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "af03de7906444e42b079c3a09be302d7"
format_stamp: "Formatted at 2026-06-19 06:58:10 on dist-test-slave-10d0"
server_key: "9bdde1a72973b2db49b767dac142b50f"
server_key_iv: "14c5816e8b91fbbda289e2e5281a5013"
server_key_version: "encryptionkey@0"
I20260619 06:58:10.182580 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "af03de7906444e42b079c3a09be302d7"
format_stamp: "Formatted at 2026-06-19 06:58:10 on dist-test-slave-10d0"
server_key: "9bdde1a72973b2db49b767dac142b50f"
server_key_iv: "14c5816e8b91fbbda289e2e5281a5013"
server_key_version: "encryptionkey@0"
I20260619 06:58:10.186196 14460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:58:10.188416 14475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:10.189620 14460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:10.189745 14460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "af03de7906444e42b079c3a09be302d7"
format_stamp: "Formatted at 2026-06-19 06:58:10 on dist-test-slave-10d0"
server_key: "9bdde1a72973b2db49b767dac142b50f"
server_key_iv: "14c5816e8b91fbbda289e2e5281a5013"
server_key_version: "encryptionkey@0"
I20260619 06:58:10.189855 14460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:10.210157 14460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:10.214643 14460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:10.214910 14460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:10.215700 14460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:10.216825 14460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:10.216885 14460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:10.216938 14460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:10.216966 14460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:10.226192 14460 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:35707
I20260619 06:58:10.226213 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:35707 every 8 connection(s)
I20260619 06:58:10.227063 14460 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:10.233346 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 14460
I20260619 06:58:10.233434 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:10.233701 11689 external_mini_cluster.cc:1428] Setting key b1f7cb8d035998f1639d4df0eb689f25
Jun 19 06:58:10 dist-test-slave-10d0 krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852290, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:10.240130 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:10.228888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:49173 (local address 127.11.106.126:46613)
0619 06:58:10.229029 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:10.229033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:10.230072 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 06:58:10.230901 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:10.230904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:10.230951 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:10.231024 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:10.232159 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:10.232767 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:10.234039 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:10.234301 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:10.234428 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:10.236883 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:10.236887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:10.236896 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:10.236920 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:10.238139 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:10.238652 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:10.238657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:10.238658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:10.238742 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:10.239082 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:10.239086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:10.239088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:10.239263 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:10.239380 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:10.239856 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 06:58:10.239974 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:58:10.241001 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:10.229299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46613 (local address 127.11.106.67:49173)
0619 06:58:10.229927 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:10.229968 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:10.230708 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:10.231190 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:10.231197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:10.231539 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:10.232029 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:10.232043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:10.232892 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:10.232895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:10.233912 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:10.233922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:10.234465 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:10.234469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:10.234645 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:10.235268 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:10.235285 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:10.236721 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:10.238259 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:10.238265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:10.238279 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:10.238545 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:10.238843 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:10.238846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:10.238848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:10.238983 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:10.239378 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:10.239385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:10.239624 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 06:58:10.239814 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 06:58:10.240134 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":99,"threads_started":1}
I20260619 06:58:10.242079 14589 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46613
I20260619 06:58:10.242311 14589 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:10.242786 14589 heartbeater.cc:511] Master 127.11.106.126:46613 requested a full tablet report, sending...
I20260619 06:58:10.243788 14126 ts_manager.cc:194] Registered new tserver with Master: af03de7906444e42b079c3a09be302d7 (127.11.106.67:35707)
I20260619 06:58:10.244308 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:49173
I20260619 06:58:10.247354 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:58:10.261566 14169 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:10.259892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.11.106.126:46613)
0619 06:58:10.260026 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:10.260030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:10.260280 (+ 250us) server_negotiation.cc:407] Connection header received
0619 06:58:10.260339 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:10.260342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:10.260388 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:10.260470 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:10.261397 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41522: BlockingRecv error: recv got EOF from 127.0.0.1:41522 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:58:10.263481 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 14192
I20260619 06:58:10.269465 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 14326
I20260619 06:58:10.275070 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 14460
I20260619 06:58:10.280463 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 14098
2026-06-19T06:58:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2170 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:10 dist-test-slave-10d0 krb5kdc[14601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:10 dist-test-slave-10d0 krb5kdc[14601](info): set up 2 sockets
Jun 19 06:58:10 dist-test-slave-10d0 krb5kdc[14601](info): commencing operation
krb5kdc: starting...
W20260619 06:58:11.349259 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:11 dist-test-slave-10d0 krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:11.476613 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46867
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:46867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:11.595628 14614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:11.595921 14614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:11.595963 14614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:11.600175 14614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:11.600250 14614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:11.600270 14614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:11.600287 14614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:11.600302 14614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:11.605960 14614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:46867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:11.607143 14614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:11.608379 14614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:11.617340 14619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:11.617274 14622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:11.617274 14620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:11.617523 14614 server_base.cc:1034] running on GCE node
I20260619 06:58:11.618178 14614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:11.619441 14614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:11.620618 14614 hybrid_clock.cc:648] HybridClock initialized: now 1781852291620603 us; error 32 us; skew 500 ppm
Jun 19 06:58:11 dist-test-slave-10d0 krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852291, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:11.627182 14614 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:11.628183 14614 webserver.cc:460] Webserver started at http://127.11.106.126:43979/ using document root <none> and password file <none>
I20260619 06:58:11.628844 14614 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:11.628937 14614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:11.629176 14614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:11.631301 14614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "0306dca244c94d71a2d30ff2e271da20"
format_stamp: "Formatted at 2026-06-19 06:58:11 on dist-test-slave-10d0"
server_key: "ace263f52e137d8009b7904d86dd2e8c"
server_key_iv: "dbccdd4f0f1f8aac5c271da3eae81b32"
server_key_version: "encryptionkey@0"
I20260619 06:58:11.631886 14614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "0306dca244c94d71a2d30ff2e271da20"
format_stamp: "Formatted at 2026-06-19 06:58:11 on dist-test-slave-10d0"
server_key: "ace263f52e137d8009b7904d86dd2e8c"
server_key_iv: "dbccdd4f0f1f8aac5c271da3eae81b32"
server_key_version: "encryptionkey@0"
I20260619 06:58:11.635780 14614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:58:11.638399 14629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:11.639631 14614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:11.639786 14614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "0306dca244c94d71a2d30ff2e271da20"
format_stamp: "Formatted at 2026-06-19 06:58:11 on dist-test-slave-10d0"
server_key: "ace263f52e137d8009b7904d86dd2e8c"
server_key_iv: "dbccdd4f0f1f8aac5c271da3eae81b32"
server_key_version: "encryptionkey@0"
I20260619 06:58:11.639909 14614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:11.660874 14614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:11.663910 14614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:11.664124 14614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:11.671115 14614 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:46867
I20260619 06:58:11.671238 14681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:46867 every 8 connection(s)
I20260619 06:58:11.672156 14614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:11.673516 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 14614
I20260619 06:58:11.673606 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:11.673871 11689 external_mini_cluster.cc:1428] Setting key 86c849df043957aa239dba67acf704a6
I20260619 06:58:11.675606 14682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:11.682804 14682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20: Bootstrap starting.
Jun 19 06:58:11 dist-test-slave-10d0 krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:11.685542 14682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:11.686355 14682 log.cc:824] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:11.688519 14682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20: No bootstrap required, opened a new log
I20260619 06:58:11.689065 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:11.675194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.11.106.126:46867)
0619 06:58:11.675631 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:11.675648 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:11.675685 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:58:11.676377 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:11.676395 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:11.676822 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:11.677104 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:11.678126 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:11.679129 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:11.680322 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:11.680863 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:11.680986 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:11.683982 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:11.683987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:11.684001 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:11.684029 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:11.685836 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:11.686290 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:11.686311 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:11.686316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:11.686424 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:11.686715 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:11.686719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:11.686721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:11.687272 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:11.687472 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:11.687781 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 06:58:11.688156 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":164,"threads_started":1}
I20260619 06:58:11.691946 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:58:11.692502 14682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0306dca244c94d71a2d30ff2e271da20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46867 } }
I20260619 06:58:11.692842 14682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:11.692941 14682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0306dca244c94d71a2d30ff2e271da20, State: Initialized, Role: FOLLOWER
I20260619 06:58:11.693400 14682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0306dca244c94d71a2d30ff2e271da20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46867 } }
I20260619 06:58:11.693568 14682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:11.693670 14682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:11.693815 14682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:11.695305 14682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0306dca244c94d71a2d30ff2e271da20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46867 } }
I20260619 06:58:11.695698 14682 leader_election.cc:302] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0306dca244c94d71a2d30ff2e271da20; no voters:
I20260619 06:58:11.696086 14682 leader_election.cc:288] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:11.696189 14688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:11.696399 14688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [term 1 LEADER]: Becoming Leader. State: Replica: 0306dca244c94d71a2d30ff2e271da20, State: Running, Role: LEADER
I20260619 06:58:11.696765 14688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0306dca244c94d71a2d30ff2e271da20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46867 } }
I20260619 06:58:11.697523 14682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:11.698007 14690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0306dca244c94d71a2d30ff2e271da20. Latest consensus state: current_term: 1 leader_uuid: "0306dca244c94d71a2d30ff2e271da20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0306dca244c94d71a2d30ff2e271da20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46867 } } }
I20260619 06:58:11.698137 14690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:11.698496 14689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0306dca244c94d71a2d30ff2e271da20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0306dca244c94d71a2d30ff2e271da20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46867 } } }
I20260619 06:58:11.698625 14689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:11.698949 14692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:11.702608 14692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:11.709383 14692 catalog_manager.cc:1347] Generated new cluster ID: 8e3767c10e534357afb446f85d0e54a7
I20260619 06:58:11.709455 14692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:11.712919 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:11.707353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.11.106.126:46867)
0619 06:58:11.707480 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:11.707485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:11.707988 (+ 503us) server_negotiation.cc:407] Connection header received
0619 06:58:11.708191 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:11.708195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:11.708244 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:11.708299 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:11.709094 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:11.709823 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:11.710707 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:11.710965 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:11.711080 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:11.711950 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:11.711954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:11.711957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:11.712392 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:11.712421 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:11.712436 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:11.712539 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:11.712703 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 06:58:11.712767 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 06:58:11.722352 14692 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:11.723271 14692 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:11.729588 14692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0306dca244c94d71a2d30ff2e271da20: Generated new TSK 0
I20260619 06:58:11.730268 14692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:58:11.740154 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 14614
2026-06-19T06:58:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1458 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:11 dist-test-slave-10d0 krb5kdc[14715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:11 dist-test-slave-10d0 krb5kdc[14715](info): set up 2 sockets
Jun 19 06:58:11 dist-test-slave-10d0 krb5kdc[14715](info): commencing operation
krb5kdc: starting...
W20260619 06:58:12.797226 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:12 dist-test-slave-10d0 krb5kdc[14715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:12.923557 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38439
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:13.035386 14728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:13.035670 14728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:13.035718 14728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:13.039732 14728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:13.039798 14728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:13.039819 14728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:13.039835 14728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:13.039851 14728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:13.045106 14728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:38439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:13.046223 14728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:13.047235 14728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:13.056005 14733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:13.055970 14736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:13.056162 14728 server_base.cc:1034] running on GCE node
W20260619 06:58:13.055982 14734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:13.056937 14728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:13.058092 14728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:13.059273 14728 hybrid_clock.cc:648] HybridClock initialized: now 1781852293059260 us; error 37 us; skew 500 ppm
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:13.062325 14728 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:13.063314 14728 webserver.cc:460] Webserver started at http://127.11.106.126:46483/ using document root <none> and password file <none>
I20260619 06:58:13.064054 14728 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:13.064177 14728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:13.064446 14728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:13.066778 14728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "b84e9571ef394b099606b99e400551f5"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "b94eed4cf3929f8a620da47cb0d0f8b8"
server_key_iv: "6450430078bd8fe4b6f5f0de12ecd80e"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.067404 14728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "b84e9571ef394b099606b99e400551f5"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "b94eed4cf3929f8a620da47cb0d0f8b8"
server_key_iv: "6450430078bd8fe4b6f5f0de12ecd80e"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.071666 14728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:58:13.074226 14743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.075487 14728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:58:13.075630 14728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "b84e9571ef394b099606b99e400551f5"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "b94eed4cf3929f8a620da47cb0d0f8b8"
server_key_iv: "6450430078bd8fe4b6f5f0de12ecd80e"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.075743 14728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:13.102876 14728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:13.106406 14728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:13.106653 14728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:13.114010 14728 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:38439
I20260619 06:58:13.114324 14795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:38439 every 8 connection(s)
I20260619 06:58:13.115185 14728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:13.118063 14796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:13.120344 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 14728
I20260619 06:58:13.120493 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:13.120870 11689 external_mini_cluster.cc:1428] Setting key 9364c766d9b8b5a048278e569afad292
I20260619 06:58:13.124598 14796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5: Bootstrap starting.
I20260619 06:58:13.127357 14796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:13.128203 14796 log.cc:824] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:13.130347 14796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5: No bootstrap required, opened a new log
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:13.133774 14796 raft_consensus.cc:348] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84e9571ef394b099606b99e400551f5" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38439 } }
I20260619 06:58:13.133996 14796 raft_consensus.cc:374] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:13.134056 14796 raft_consensus.cc:729] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84e9571ef394b099606b99e400551f5, State: Initialized, Role: FOLLOWER
I20260619 06:58:13.134514 14796 consensus_queue.cc:260] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84e9571ef394b099606b99e400551f5" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38439 } }
I20260619 06:58:13.134693 14796 raft_consensus.cc:388] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:13.134769 14796 raft_consensus.cc:482] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:13.134881 14796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:13.135964 14796 raft_consensus.cc:504] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84e9571ef394b099606b99e400551f5" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38439 } }
I20260619 06:58:13.136369 14796 leader_election.cc:302] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b84e9571ef394b099606b99e400551f5; no voters:
I20260619 06:58:13.136754 14796 leader_election.cc:288] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:13.136848 14801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:13.137075 14801 raft_consensus.cc:686] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [term 1 LEADER]: Becoming Leader. State: Replica: b84e9571ef394b099606b99e400551f5, State: Running, Role: LEADER
I20260619 06:58:13.137473 14801 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84e9571ef394b099606b99e400551f5" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38439 } }
I20260619 06:58:13.137871 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:13.122390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.11.106.126:38439)
0619 06:58:13.122850 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:13.122870 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:13.122914 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:58:13.123754 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:13.123793 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:13.124229 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:13.124632 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:13.125447 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.126943 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.127947 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.128680 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.128812 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:13.131753 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:13.131758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:13.131772 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:13.131799 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:13.134502 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.135022 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.135028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.135033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.135154 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.135472 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.135476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.135477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.135941 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:13.136136 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:13.136390 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 06:58:13.136738 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":121,"threads_started":1}
I20260619 06:58:13.138164 14796 sys_catalog.cc:564] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:13.138754 14801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b84e9571ef394b099606b99e400551f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84e9571ef394b099606b99e400551f5" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38439 } } }
I20260619 06:58:13.138859 14801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:13.139223 14802 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84e9571ef394b099606b99e400551f5. Latest consensus state: current_term: 1 leader_uuid: "b84e9571ef394b099606b99e400551f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84e9571ef394b099606b99e400551f5" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38439 } } }
I20260619 06:58:13.139320 14802 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:13.140416 14807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:13.143198 14807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:13.149057 14807 catalog_manager.cc:1347] Generated new cluster ID: d1a2de99bc0b4d22b3fb56bbcebf8bb4
I20260619 06:58:13.149134 14807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:13.161672 14807 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:13.162631 14807 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:13.173962 14807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b84e9571ef394b099606b99e400551f5: Generated new TSK 0
I20260619 06:58:13.174700 14807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:58:13.214988 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38439
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:13.334815 14822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:13.335074 14822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:13.335117 14822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:13.339260 14822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:13.339322 14822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:13.339427 14822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:13.344810 14822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:13.345916 14822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:13.346966 14822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:13.356611 14830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:13.356729 14827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:13.356652 14828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:13.356804 14822 server_base.cc:1034] running on GCE node
I20260619 06:58:13.357322 14822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:13.357926 14822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:13.359179 14822 hybrid_clock.cc:648] HybridClock initialized: now 1781852293359144 us; error 48 us; skew 500 ppm
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:13.362213 14822 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:58:13.363076 14822 webserver.cc:460] Webserver started at http://127.11.106.65:34099/ using document root <none> and password file <none>
I20260619 06:58:13.363656 14822 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:13.363750 14822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:13.364003 14822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:13.366142 14822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "58c2ba4fb8a44bad89f800e1e52b28c6"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "937980781989e0282590266e67f4d9ab"
server_key_iv: "cba6ab74d158a82b776b54228b2024fd"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.366698 14822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "58c2ba4fb8a44bad89f800e1e52b28c6"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "937980781989e0282590266e67f4d9ab"
server_key_iv: "cba6ab74d158a82b776b54228b2024fd"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.370525 14822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 06:58:13.373019 14837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.374269 14822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:13.374413 14822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "58c2ba4fb8a44bad89f800e1e52b28c6"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "937980781989e0282590266e67f4d9ab"
server_key_iv: "cba6ab74d158a82b776b54228b2024fd"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.374526 14822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:13.387060 14822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:13.390106 14822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:13.390321 14822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:13.390959 14822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:13.391820 14822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:13.391904 14822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.391990 14822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:13.392041 14822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.402182 14822 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:38973
I20260619 06:58:13.402212 14950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:38973 every 8 connection(s)
I20260619 06:58:13.403113 14822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:13.412478 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 14822
I20260619 06:58:13.412672 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:13.412984 11689 external_mini_cluster.cc:1428] Setting key b953aa5233a3ca020fba0c444ddef381
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:13.417567 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:13.404998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:38537 (local address 127.11.106.126:38439)
0619 06:58:13.405185 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:13.405191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:13.406054 (+ 863us) server_negotiation.cc:407] Connection header received
0619 06:58:13.407476 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:13.407478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:13.407526 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:13.407603 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:13.408869 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.409417 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.410545 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.410758 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.410835 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:13.413745 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:13.413749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:13.413751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:13.413775 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:13.415333 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.415893 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.415897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.415899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.415996 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.416368 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.416372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.416373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.416639 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:13.416746 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:13.417257 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 06:58:13.417397 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:58:13.418435 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:13.405432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38439 (local address 127.11.106.65:38537)
0619 06:58:13.405900 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:13.405944 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:13.407277 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:13.407779 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:13.407786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:13.408141 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:13.408727 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:13.408743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.409535 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.409537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:13.410434 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:13.410444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.410886 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.410888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:13.411037 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:13.411669 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:13.411687 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:13.413551 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:13.415468 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:13.415475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:13.415491 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:13.415767 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:13.416101 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:13.416107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:13.416111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:13.416262 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:13.416744 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:13.416752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:13.417014 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 06:58:13.417216 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 06:58:13.417555 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":119,"threads_started":1}
I20260619 06:58:13.419757 14951 heartbeater.cc:347] Connected to a master server at 127.11.106.126:38439
I20260619 06:58:13.420030 14951 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:13.420614 14951 heartbeater.cc:511] Master 127.11.106.126:38439 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 06:58:13.422394 14760 ts_manager.cc:194] Registered new tserver with Master: 58c2ba4fb8a44bad89f800e1e52b28c6 (127.11.106.65:38973)
I20260619 06:58:13.423674 14760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:38537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:58:13.440510 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38439
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:13.552462 14956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:13.552904 14956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:13.552968 14956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:13.556980 14956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:13.557036 14956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:13.557094 14956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:13.562371 14956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:13.563472 14956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:13.564476 14956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:13.574374 14962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:13.574386 14961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:13.574388 14964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:13.574486 14956 server_base.cc:1034] running on GCE node
I20260619 06:58:13.575122 14956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:13.575790 14956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:13.576989 14956 hybrid_clock.cc:648] HybridClock initialized: now 1781852293576952 us; error 49 us; skew 500 ppm
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:13.579793 14956 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:58:13.580770 14956 webserver.cc:460] Webserver started at http://127.11.106.66:46695/ using document root <none> and password file <none>
I20260619 06:58:13.581346 14956 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:13.581440 14956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:13.581681 14956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:13.583765 14956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "3a5c1b016792493dbadc41ce92091ac5"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "ae3bce71796ed9022e0f1fa40f63d573"
server_key_iv: "700ddde37ab0650becb9a4e240f9340f"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.584342 14956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "3a5c1b016792493dbadc41ce92091ac5"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "ae3bce71796ed9022e0f1fa40f63d573"
server_key_iv: "700ddde37ab0650becb9a4e240f9340f"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.588121 14956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:58:13.590682 14971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.592032 14956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:58:13.592188 14956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "3a5c1b016792493dbadc41ce92091ac5"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "ae3bce71796ed9022e0f1fa40f63d573"
server_key_iv: "700ddde37ab0650becb9a4e240f9340f"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.592306 14956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:13.614235 14956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:13.617424 14956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:13.617647 14956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:13.618258 14956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:13.619112 14956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:13.619184 14956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.619256 14956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:13.619310 14956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.628496 14956 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:42075
I20260619 06:58:13.628520 15084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:42075 every 8 connection(s)
I20260619 06:58:13.629472 14956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:13.637082 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 14956
I20260619 06:58:13.637218 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:13.637553 11689 external_mini_cluster.cc:1428] Setting key 8411e45b5344f3280425358e2549ff59
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:13.645485 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:13.631272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:49657 (local address 127.11.106.126:38439)
0619 06:58:13.631446 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:13.631452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:13.632327 (+ 875us) server_negotiation.cc:407] Connection header received
0619 06:58:13.633731 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:13.633733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:13.633785 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:13.633861 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:13.635031 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.635559 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.636798 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.637197 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.637324 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:13.641243 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:13.641246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:13.641248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:13.641297 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:13.643001 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.643681 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.643683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.643685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.643756 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.644168 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.644170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.644171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.644426 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:13.644506 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:13.645256 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 06:58:13.645362 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:58:13.646867 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:13.631665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38439 (local address 127.11.106.66:49657)
0619 06:58:13.632160 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:13.632205 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:13.633525 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:13.634032 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:13.634039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:13.634411 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:13.634887 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:13.634901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.635705 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.635708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:13.636685 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:13.636695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.637397 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.637403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:13.637618 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:13.638482 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:13.638504 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:13.641049 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:13.643134 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:13.643143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:13.643165 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:13.643562 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:13.643846 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:13.643852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:13.643857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:13.644065 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:13.644577 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:13.644588 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:13.644952 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 06:58:13.645189 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 06:58:13.645623 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":124,"threads_started":1}
I20260619 06:58:13.648236 15085 heartbeater.cc:347] Connected to a master server at 127.11.106.126:38439
I20260619 06:58:13.648497 15085 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 06:58:13.649089 15085 heartbeater.cc:511] Master 127.11.106.126:38439 requested a full tablet report, sending...
I20260619 06:58:13.650271 14760 ts_manager.cc:194] Registered new tserver with Master: 3a5c1b016792493dbadc41ce92091ac5 (127.11.106.66:42075)
I20260619 06:58:13.651000 14760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:49657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:58:13.668349 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38439
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:13.782810 15090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:13.783070 15090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:13.783119 15090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:13.787138 15090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:13.787199 15090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:13.787259 15090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:13.792448 15090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:13.793485 15090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:13.794538 15090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:13.804298 15095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:13.804324 15096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:13.804327 15098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:13.804391 15090 server_base.cc:1034] running on GCE node
I20260619 06:58:13.804958 15090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:13.805663 15090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:13.806866 15090 hybrid_clock.cc:648] HybridClock initialized: now 1781852293806845 us; error 37 us; skew 500 ppm
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:13.809886 15090 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:58:13.810819 15090 webserver.cc:460] Webserver started at http://127.11.106.67:35553/ using document root <none> and password file <none>
I20260619 06:58:13.811430 15090 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:13.811523 15090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:13.811769 15090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:13.814071 15090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "6db35ebe6b1541a8a51eb9dd2bfb3c61"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "dbef5558ca127e62149d8ce2322243a7"
server_key_iv: "e44836ef25a475eb4c22374e7bfdf935"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.814733 15090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "6db35ebe6b1541a8a51eb9dd2bfb3c61"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "dbef5558ca127e62149d8ce2322243a7"
server_key_iv: "e44836ef25a475eb4c22374e7bfdf935"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.818861 15090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:13.821593 15105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.823051 15090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:58:13.823211 15090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "6db35ebe6b1541a8a51eb9dd2bfb3c61"
format_stamp: "Formatted at 2026-06-19 06:58:13 on dist-test-slave-10d0"
server_key: "dbef5558ca127e62149d8ce2322243a7"
server_key_iv: "e44836ef25a475eb4c22374e7bfdf935"
server_key_version: "encryptionkey@0"
I20260619 06:58:13.823341 15090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:13.856725 15090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:13.860527 15090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:13.860816 15090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:13.861496 15090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:13.862416 15090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:13.862493 15090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.862568 15090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:13.862613 15090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:13.871582 15090 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:36873
I20260619 06:58:13.871608 15218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:36873 every 8 connection(s)
I20260619 06:58:13.872529 15090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:13.875375 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 15090
I20260619 06:58:13.875500 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:13.875820 11689 external_mini_cluster.cc:1428] Setting key f1c57f72e03854483eb7a6c81808698d
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852293, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:13.887909 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:13.874546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:48389 (local address 127.11.106.126:38439)
0619 06:58:13.874708 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:13.874712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:13.875563 (+ 851us) server_negotiation.cc:407] Connection header received
0619 06:58:13.877373 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:13.877376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:13.877433 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:13.877542 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:13.878914 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.879543 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.880884 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.881184 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.881301 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:13.884463 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:13.884467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:13.884470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:13.884498 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:13.885780 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.886337 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.886342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.886346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.886441 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.886788 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.886793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.886795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.886991 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:13.887102 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:13.887603 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 06:58:13.887725 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:58:13.888935 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:13.874899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38439 (local address 127.11.106.67:48389)
0619 06:58:13.875378 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:13.875422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:13.877144 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:13.877726 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:13.877733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:13.878269 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:13.878768 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:13.878781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.879667 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.879670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:13.880756 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:13.880766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.881365 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.881370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:13.881571 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:13.882244 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:13.882269 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:13.884288 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:13.885930 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:13.885936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:13.885952 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:13.886216 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:13.886541 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:13.886544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:13.886546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:13.886691 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:13.887098 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:13.887105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:13.887367 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 06:58:13.887560 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 06:58:13.887884 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":139,"threads_started":1}
I20260619 06:58:13.890173 15219 heartbeater.cc:347] Connected to a master server at 127.11.106.126:38439
I20260619 06:58:13.890439 15219 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:13.890983 15219 heartbeater.cc:511] Master 127.11.106.126:38439 requested a full tablet report, sending...
I20260619 06:58:13.892067 14760 ts_manager.cc:194] Registered new tserver with Master: 6db35ebe6b1541a8a51eb9dd2bfb3c61 (127.11.106.67:36873)
I20260619 06:58:13.892743 14760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:48389
I20260619 06:58:13.902022 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852293, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 06:58:13 dist-test-slave-10d0 krb5kdc[14715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852293, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:13.928676 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:13.919690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.11.106.126:38439)
0619 06:58:13.919932 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:13.919937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:13.920086 (+ 149us) server_negotiation.cc:407] Connection header received
0619 06:58:13.920270 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:13.920285 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:13.920340 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:13.920439 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:13.921212 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.921744 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.922611 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:13.922839 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:13.922951 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:13.925493 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:13.925496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:13.925498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:13.925523 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:13.926932 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.927309 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.927313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.927315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.927391 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:13.927684 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:13.927687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:13.927692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:13.927976 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:13.928083 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:13.928295 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 06:58:13.928397 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
W20260619 06:58:13.929086 14760 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37988
I20260619 06:58:13.930032 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 14822
I20260619 06:58:13.936261 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 14956
I20260619 06:58:13.942646 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 15090
I20260619 06:58:13.948165 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 14728
2026-06-19T06:58:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2209 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:14 dist-test-slave-10d0 krb5kdc[15231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:14 dist-test-slave-10d0 krb5kdc[15231](info): set up 2 sockets
Jun 19 06:58:14 dist-test-slave-10d0 krb5kdc[15231](info): commencing operation
krb5kdc: starting...
W20260619 06:58:15.001363 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:15 dist-test-slave-10d0 krb5kdc[15231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:15.129684 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:41323
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:41323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:15.245550 15244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:15.245916 15244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:15.246001 15244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:15.250284 15244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:15.250393 15244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:15.250433 15244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:15.250489 15244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:15.250526 15244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:15.256194 15244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:41323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:41323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:15.257448 15244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:15.258661 15244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:15.268083 15249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:15.268187 15244 server_base.cc:1034] running on GCE node
W20260619 06:58:15.268083 15250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:15.268144 15252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:15.269106 15244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:15.270392 15244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:15.271628 15244 hybrid_clock.cc:648] HybridClock initialized: now 1781852295271603 us; error 43 us; skew 500 ppm
Jun 19 06:58:15 dist-test-slave-10d0 krb5kdc[15231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:15.274765 15244 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:15.275748 15244 webserver.cc:460] Webserver started at http://127.11.106.126:36047/ using document root <none> and password file <none>
I20260619 06:58:15.276371 15244 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:15.276434 15244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:15.276685 15244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:15.278882 15244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "f2b34d664e224583a9223c19a9f4bedd"
format_stamp: "Formatted at 2026-06-19 06:58:15 on dist-test-slave-10d0"
server_key: "92a238cdf941fea2406a8fb21d76b766"
server_key_iv: "bfbbf8cac18a4da3ec5c05e24176bbba"
server_key_version: "encryptionkey@0"
I20260619 06:58:15.279417 15244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "f2b34d664e224583a9223c19a9f4bedd"
format_stamp: "Formatted at 2026-06-19 06:58:15 on dist-test-slave-10d0"
server_key: "92a238cdf941fea2406a8fb21d76b766"
server_key_iv: "bfbbf8cac18a4da3ec5c05e24176bbba"
server_key_version: "encryptionkey@0"
I20260619 06:58:15.283445 15244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:15.286394 15259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:15.287887 15244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:58:15.288018 15244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "f2b34d664e224583a9223c19a9f4bedd"
format_stamp: "Formatted at 2026-06-19 06:58:15 on dist-test-slave-10d0"
server_key: "92a238cdf941fea2406a8fb21d76b766"
server_key_iv: "bfbbf8cac18a4da3ec5c05e24176bbba"
server_key_version: "encryptionkey@0"
I20260619 06:58:15.288129 15244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:15.311108 15244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:15.314719 15244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:15.314911 15244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:15.323129 15244 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:41323
I20260619 06:58:15.323215 15311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:41323 every 8 connection(s)
I20260619 06:58:15.324239 15244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:15.326984 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 15244
I20260619 06:58:15.327095 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:15.327416 11689 external_mini_cluster.cc:1428] Setting key b88812e7d36bd4886a40a598375c9d4c
I20260619 06:58:15.327507 15312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:15.335247 15312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd: Bootstrap starting.
Jun 19 06:58:15 dist-test-slave-10d0 krb5kdc[15231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:15.338371 15312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:15.339172 15312 log.cc:824] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:15.341722 15312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd: No bootstrap required, opened a new log
I20260619 06:58:15.343916 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:15.329161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.11.106.126:41323)
0619 06:58:15.329653 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:15.329671 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:15.329715 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:58:15.330416 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:15.330436 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:15.330822 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:15.331162 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:15.332194 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.333344 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.334575 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.335094 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.335242 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:15.338401 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:15.338407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:15.338478 (+ 71us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:15.338525 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:15.340620 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:15.341101 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:15.341108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:15.341114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:15.341257 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:15.341543 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:15.341546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:15.341548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:15.342101 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:15.342309 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:15.342570 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 06:58:15.342900 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":152,"threads_started":1}
I20260619 06:58:15.346341 15312 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b34d664e224583a9223c19a9f4bedd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41323 } }
I20260619 06:58:15.346653 15312 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:15.346733 15312 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b34d664e224583a9223c19a9f4bedd, State: Initialized, Role: FOLLOWER
I20260619 06:58:15.347167 15312 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b34d664e224583a9223c19a9f4bedd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41323 } }
I20260619 06:58:15.347316 15312 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:15.347383 15312 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:15.347484 15312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:15.348472 15312 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b34d664e224583a9223c19a9f4bedd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41323 } }
I20260619 06:58:15.348862 15312 leader_election.cc:302] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2b34d664e224583a9223c19a9f4bedd; no voters:
I20260619 06:58:15.349236 15312 leader_election.cc:288] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:15.349385 15317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:15.349650 15317 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [term 1 LEADER]: Becoming Leader. State: Replica: f2b34d664e224583a9223c19a9f4bedd, State: Running, Role: LEADER
I20260619 06:58:15.350023 15317 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b34d664e224583a9223c19a9f4bedd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41323 } }
I20260619 06:58:15.350551 15312 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:15.351878 15319 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2b34d664e224583a9223c19a9f4bedd. Latest consensus state: current_term: 1 leader_uuid: "f2b34d664e224583a9223c19a9f4bedd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b34d664e224583a9223c19a9f4bedd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41323 } } }
I20260619 06:58:15.352005 15319 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:15.352622 15318 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2b34d664e224583a9223c19a9f4bedd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2b34d664e224583a9223c19a9f4bedd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41323 } } }
I20260619 06:58:15.352726 15318 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:15.352871 15326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:15.355587 15326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:15.362018 15326 catalog_manager.cc:1347] Generated new cluster ID: 6665aae55c244b16884a7d13e9777a9b
I20260619 06:58:15.362125 15326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:15.370168 15326 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:15.371582 15326 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:15.377059 15326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2b34d664e224583a9223c19a9f4bedd: Generated new TSK 0
I20260619 06:58:15.377918 15326 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:58:15.417338 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:41323
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:15.563397 15338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:15.563740 15338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:15.563791 15338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:15.568697 15338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:15.568781 15338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:15.568847 15338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:15.574846 15338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:41323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:15.576144 15338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:15.577314 15338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:58:15.588418 15338 server_base.cc:1034] running on GCE node
W20260619 06:58:15.588305 15343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:15.588332 15346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:15.588311 15344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:15.589068 15338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:15.589845 15338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:15.591051 15338 hybrid_clock.cc:648] HybridClock initialized: now 1781852295591005 us; error 60 us; skew 500 ppm
Jun 19 06:58:15 dist-test-slave-10d0 krb5kdc[15231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:15.594236 15338 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:58:15.595242 15338 webserver.cc:460] Webserver started at http://127.11.106.65:34913/ using document root <none> and password file <none>
I20260619 06:58:15.595891 15338 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:15.595991 15338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:15.596247 15338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:15.598619 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "4f73dd02205343599e0f6d3af53a755c"
format_stamp: "Formatted at 2026-06-19 06:58:15 on dist-test-slave-10d0"
server_key: "f399e88332c58fd4ed790afda88de9a9"
server_key_iv: "658579f8b07663766fa393c4a232b2fc"
server_key_version: "encryptionkey@0"
I20260619 06:58:15.599257 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "4f73dd02205343599e0f6d3af53a755c"
format_stamp: "Formatted at 2026-06-19 06:58:15 on dist-test-slave-10d0"
server_key: "f399e88332c58fd4ed790afda88de9a9"
server_key_iv: "658579f8b07663766fa393c4a232b2fc"
server_key_version: "encryptionkey@0"
I20260619 06:58:15.603695 15338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:58:15.606700 15353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:15.608318 15338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 06:58:15.608500 15338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "4f73dd02205343599e0f6d3af53a755c"
format_stamp: "Formatted at 2026-06-19 06:58:15 on dist-test-slave-10d0"
server_key: "f399e88332c58fd4ed790afda88de9a9"
server_key_iv: "658579f8b07663766fa393c4a232b2fc"
server_key_version: "encryptionkey@0"
I20260619 06:58:15.608716 15338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:15.631333 15338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:15.635453 15338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:15.635681 15338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:15.636358 15338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:15.637358 15338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:15.637447 15338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:15.637544 15338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:15.637596 15338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:15.647235 15338 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:34965
I20260619 06:58:15.647255 15466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:34965 every 8 connection(s)
I20260619 06:58:15.648303 15338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:15.656621 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 15338
I20260619 06:58:15.656755 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:15.657092 11689 external_mini_cluster.cc:1428] Setting key d9b3c2a918efa5fec75320d782a7c383
Jun 19 06:58:15 dist-test-slave-10d0 krb5kdc[15231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:15.665248 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:15.650495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:34319 (local address 127.11.106.126:41323)
0619 06:58:15.650675 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:15.650679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:15.651793 (+ 1114us) server_negotiation.cc:407] Connection header received
0619 06:58:15.653494 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:15.653498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:15.653553 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:15.653702 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:15.655019 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.655594 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.657198 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.657460 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.657603 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:15.660968 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:15.660972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:15.660974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:15.661002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:15.662600 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:15.663353 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:15.663356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:15.663358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:15.663446 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:15.663896 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:15.663898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:15.663899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:15.664091 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:15.664167 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:15.664992 (+ 825us) server_negotiation.cc:299] Negotiation successful
0619 06:58:15.665117 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:58:15.666620 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:15.650975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:41323 (local address 127.11.106.65:34319)
0619 06:58:15.651592 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:15.651650 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:15.653246 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:15.653904 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:15.653912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:15.654319 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:15.654845 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:15.654859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.655755 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.655758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:15.657030 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:15.657044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.657659 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.657662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:15.657905 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:15.658664 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:15.658693 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:15.660756 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:15.662755 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:15.662766 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:15.662791 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:15.663220 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:15.663548 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:15.663553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:15.663557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:15.663788 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:15.664207 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:15.664217 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:15.664658 (+ 441us) client_negotiation.cc:769] Sending connection context
0619 06:58:15.664932 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 06:58:15.665359 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":238,"threads_started":1}
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 06:58:15.668174 15467 heartbeater.cc:347] Connected to a master server at 127.11.106.126:41323
I20260619 06:58:15.668478 15467 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:15.669164 15467 heartbeater.cc:511] Master 127.11.106.126:41323 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 06:58:15.671020 15276 ts_manager.cc:194] Registered new tserver with Master: 4f73dd02205343599e0f6d3af53a755c (127.11.106.65:34965)
I20260619 06:58:15.672240 15276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:34319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:58:15.688181 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:41323
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:15.803073 15472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:15.803368 15472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:15.803414 15472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:15.807552 15472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:15.807626 15472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:15.807694 15472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:15.813364 15472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:41323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:15.814670 15472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:15.816076 15472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:15.826246 15478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:15.826257 15477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:15.826570 15472 server_base.cc:1034] running on GCE node
W20260619 06:58:15.826303 15480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:15.827026 15472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:15.827718 15472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:15.828908 15472 hybrid_clock.cc:648] HybridClock initialized: now 1781852295828871 us; error 48 us; skew 500 ppm
Jun 19 06:58:15 dist-test-slave-10d0 krb5kdc[15231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:15.831739 15472 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:58:15.832707 15472 webserver.cc:460] Webserver started at http://127.11.106.66:39667/ using document root <none> and password file <none>
I20260619 06:58:15.833310 15472 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:15.833405 15472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:15.833648 15472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:15.835832 15472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "8c94093719b547b5b4c8e7128bf681b4"
format_stamp: "Formatted at 2026-06-19 06:58:15 on dist-test-slave-10d0"
server_key: "3fbd93804969e76e5f5241b39ba7a7f6"
server_key_iv: "2701e98a307a335de368a75ef6981b2a"
server_key_version: "encryptionkey@0"
I20260619 06:58:15.836426 15472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "8c94093719b547b5b4c8e7128bf681b4"
format_stamp: "Formatted at 2026-06-19 06:58:15 on dist-test-slave-10d0"
server_key: "3fbd93804969e76e5f5241b39ba7a7f6"
server_key_iv: "2701e98a307a335de368a75ef6981b2a"
server_key_version: "encryptionkey@0"
I20260619 06:58:15.840201 15472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:58:15.842772 15487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:15.844028 15472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:15.844127 15472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "8c94093719b547b5b4c8e7128bf681b4"
format_stamp: "Formatted at 2026-06-19 06:58:15 on dist-test-slave-10d0"
server_key: "3fbd93804969e76e5f5241b39ba7a7f6"
server_key_iv: "2701e98a307a335de368a75ef6981b2a"
server_key_version: "encryptionkey@0"
I20260619 06:58:15.844213 15472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:15.869328 15472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:15.872977 15472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:15.873140 15472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:15.873713 15472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:15.874536 15472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:15.874584 15472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:15.874625 15472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:15.874640 15472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:15.883378 15472 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:33131
I20260619 06:58:15.883397 15600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:33131 every 8 connection(s)
I20260619 06:58:15.884305 15472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:15.885195 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 15472
I20260619 06:58:15.885281 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:15.885567 11689 external_mini_cluster.cc:1428] Setting key 1597b9aa6343cd4475786b99b18d8ddc
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
Jun 19 06:58:15 dist-test-slave-10d0 krb5kdc[15231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852295, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 06:58:15.899948 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:15.886331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:47867 (local address 127.11.106.126:41323)
0619 06:58:15.886464 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:15.886468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:15.887483 (+ 1015us) server_negotiation.cc:407] Connection header received
0619 06:58:15.889101 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:15.889105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:15.889158 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:15.889249 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:15.891130 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.892076 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.893420 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.893664 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.893786 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:15.896250 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:15.896254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:15.896259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:15.896290 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:15.897710 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:15.898331 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:15.898347 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:15.898349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:15.898434 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:15.898787 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:15.898790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:15.898792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:15.898982 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:15.899092 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:15.899716 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 06:58:15.899826 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:58:15.901117 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:15.886829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:41323 (local address 127.11.106.66:47867)
0619 06:58:15.887343 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:15.887387 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:15.888843 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:15.889605 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:15.889615 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:15.890188 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:15.890936 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:15.890957 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.892234 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.892237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:15.893289 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:15.893299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:15.893821 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:15.893825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:15.894008 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:15.894615 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:15.894632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:15.896079 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:15.897858 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:15.897865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:15.897883 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:15.898209 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:15.898530 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:15.898534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:15.898536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:15.898683 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:15.899092 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:15.899099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:15.899419 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 06:58:15.899659 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 06:58:15.900038 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":143,"threads_started":1}
I20260619 06:58:15.902400 15601 heartbeater.cc:347] Connected to a master server at 127.11.106.126:41323
I20260619 06:58:15.902689 15601 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:15.903363 15601 heartbeater.cc:511] Master 127.11.106.126:41323 requested a full tablet report, sending...
I20260619 06:58:15.904531 15276 ts_manager.cc:194] Registered new tserver with Master: 8c94093719b547b5b4c8e7128bf681b4 (127.11.106.66:33131)
I20260619 06:58:15.905164 15276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:47867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:58:15.913954 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:41323
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:16.035995 15606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:16.036294 15606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:16.036340 15606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:16.040807 15606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:16.040875 15606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:16.040972 15606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:16.046614 15606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:41323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:16.047885 15606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:16.049067 15606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:16.059684 15614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:16.059700 15611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:16.059685 15612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:16.060010 15606 server_base.cc:1034] running on GCE node
I20260619 06:58:16.060611 15606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:16.061327 15606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:16.062520 15606 hybrid_clock.cc:648] HybridClock initialized: now 1781852296062492 us; error 46 us; skew 500 ppm
Jun 19 06:58:16 dist-test-slave-10d0 krb5kdc[15231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852296, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:16.065454 15606 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:58:16.066377 15606 webserver.cc:460] Webserver started at http://127.11.106.67:42955/ using document root <none> and password file <none>
I20260619 06:58:16.066965 15606 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:16.067027 15606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:16.067261 15606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:16.069578 15606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "451e59fda460437ca1c36f54e063860b"
format_stamp: "Formatted at 2026-06-19 06:58:16 on dist-test-slave-10d0"
server_key: "a0af01137663b849738d8aa1e0e02cc9"
server_key_iv: "bb158f111bdf4b8226a9e025b3932eb5"
server_key_version: "encryptionkey@0"
I20260619 06:58:16.070185 15606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "451e59fda460437ca1c36f54e063860b"
format_stamp: "Formatted at 2026-06-19 06:58:16 on dist-test-slave-10d0"
server_key: "a0af01137663b849738d8aa1e0e02cc9"
server_key_iv: "bb158f111bdf4b8226a9e025b3932eb5"
server_key_version: "encryptionkey@0"
I20260619 06:58:16.074317 15606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:16.077100 15621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:16.078429 15606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:16.078560 15606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "451e59fda460437ca1c36f54e063860b"
format_stamp: "Formatted at 2026-06-19 06:58:16 on dist-test-slave-10d0"
server_key: "a0af01137663b849738d8aa1e0e02cc9"
server_key_iv: "bb158f111bdf4b8226a9e025b3932eb5"
server_key_version: "encryptionkey@0"
I20260619 06:58:16.078676 15606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:16.093175 15606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:16.096724 15606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:16.096942 15606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:16.097584 15606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:16.098470 15606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:16.098519 15606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:16.098559 15606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:16.098600 15606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:16.107856 15606 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:41123
I20260619 06:58:16.107874 15734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:41123 every 8 connection(s)
I20260619 06:58:16.108896 15606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:16.111366 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 15606
I20260619 06:58:16.111496 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:16.111836 11689 external_mini_cluster.cc:1428] Setting key 8a852b395c49926359a7a08bcaca06e3
Jun 19 06:58:16 dist-test-slave-10d0 krb5kdc[15231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852296, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:16.123695 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:16.110946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:39153 (local address 127.11.106.126:41323)
0619 06:58:16.111209 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:16.111215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:16.112093 (+ 878us) server_negotiation.cc:407] Connection header received
0619 06:58:16.113871 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:16.113875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:16.113937 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:16.114018 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:16.115211 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:16.115754 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:16.117013 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:16.117249 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:16.117374 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:16.120028 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:16.120032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:16.120034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:16.120069 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:16.121584 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:16.122110 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:16.122115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:16.122117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:16.122204 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:16.122567 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:16.122572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:16.122574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:16.122769 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:16.122898 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:16.123403 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 06:58:16.123529 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:58:16.124647 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:16.111336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:41323 (local address 127.11.106.67:39153)
0619 06:58:16.111908 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:16.111964 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:16.113652 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:16.114200 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:16.114207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:16.114581 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:16.115068 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:16.115082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:16.115893 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:16.115896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:16.116887 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:16.116897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:16.117412 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:16.117417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:16.117590 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:16.118243 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:16.118260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:16.119857 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:16.121707 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:16.121713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:16.121730 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:16.122001 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:16.122308 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:16.122313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:16.122315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:16.122459 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:16.122898 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:16.122906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:16.123163 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 06:58:16.123363 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 06:58:16.123701 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":163,"threads_started":1}
I20260619 06:58:16.125927 15735 heartbeater.cc:347] Connected to a master server at 127.11.106.126:41323
I20260619 06:58:16.126194 15735 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:16.126714 15735 heartbeater.cc:511] Master 127.11.106.126:41323 requested a full tablet report, sending...
I20260619 06:58:16.127748 15276 ts_manager.cc:194] Registered new tserver with Master: 451e59fda460437ca1c36f54e063860b (127.11.106.67:41123)
I20260619 06:58:16.128392 15276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:39153
I20260619 06:58:16.138396 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:58:16 dist-test-slave-10d0 krb5kdc[15231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:58:16 dist-test-slave-10d0 krb5kdc[15231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:58:16.166000 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:16.156083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.11.106.65:34965)
0619 06:58:16.156405 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:16.156423 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:16.156454 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:58:16.156516 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:16.156532 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:16.156812 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:16.156974 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:16.157723 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:16.158530 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:16.159405 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:16.159803 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:16.159928 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:16.162438 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:16.162447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:16.162462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:16.162491 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:16.164024 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:16.164494 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:16.164500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:16.164501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:16.164639 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:16.164948 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:16.164952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:16.164954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:16.165210 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:16.165473 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:16.165625 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 06:58:16.165798 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
Jun 19 06:58:16 dist-test-slave-10d0 krb5kdc[15231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:16.178197 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:16.169409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.11.106.126:41323)
0619 06:58:16.169570 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:16.169576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:16.169669 (+ 93us) server_negotiation.cc:407] Connection header received
0619 06:58:16.169843 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:16.169846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:16.169905 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:16.169994 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:16.170641 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:16.171168 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:16.171984 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:16.172239 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:16.172363 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:16.175026 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:16.175029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:16.175031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:16.175062 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:16.176418 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:16.176878 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:16.176882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:16.176884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:16.176968 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:16.177243 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:16.177248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:16.177250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:16.177493 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:16.177637 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:16.177891 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 06:58:16.178025 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 06:58:16.178663 15276 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60148
I20260619 06:58:16.179322 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 15338
I20260619 06:58:16.186376 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 15472
I20260619 06:58:16.192462 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 15606
I20260619 06:58:16.198240 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 15244
2026-06-19T06:58:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2250 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:16 dist-test-slave-10d0 krb5kdc[15747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:16 dist-test-slave-10d0 krb5kdc[15747](info): set up 2 sockets
Jun 19 06:58:16 dist-test-slave-10d0 krb5kdc[15747](info): commencing operation
krb5kdc: starting...
W20260619 06:58:17.245321 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:17 dist-test-slave-10d0 krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:17.371872 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45101
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:44063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:17.483729 15760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:17.484054 15760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:17.484141 15760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:17.488179 15760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:17.488278 15760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:17.488313 15760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:17.488350 15760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:17.488389 15760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:17.493705 15760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:44063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:45101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:17.494793 15760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:17.495863 15760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:17.504151 15765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:17.504209 15768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:17.504168 15766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:17.504441 15760 server_base.cc:1034] running on GCE node
I20260619 06:58:17.505035 15760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:17.506145 15760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:17.507341 15760 hybrid_clock.cc:648] HybridClock initialized: now 1781852297507325 us; error 27 us; skew 500 ppm
Jun 19 06:58:17 dist-test-slave-10d0 krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:17.514046 15760 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:17.514851 15760 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:17.514952 15760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:17.515204 15760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:17.517408 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "0911fb843aaf48b4a0e9d1d24ac53094"
format_stamp: "Formatted at 2026-06-19 06:58:17 on dist-test-slave-10d0"
server_key: "3b348bf4d4bdae47ed196fe95ccb70c0"
server_key_iv: "5bc4b69d8453fd809ba3723bc52d589b"
server_key_version: "encryptionkey@0"
I20260619 06:58:17.517985 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "0911fb843aaf48b4a0e9d1d24ac53094"
format_stamp: "Formatted at 2026-06-19 06:58:17 on dist-test-slave-10d0"
server_key: "3b348bf4d4bdae47ed196fe95ccb70c0"
server_key_iv: "5bc4b69d8453fd809ba3723bc52d589b"
server_key_version: "encryptionkey@0"
I20260619 06:58:17.521875 15760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:58:17.524274 15774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:17.525697 15760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:17.525856 15760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "0911fb843aaf48b4a0e9d1d24ac53094"
format_stamp: "Formatted at 2026-06-19 06:58:17 on dist-test-slave-10d0"
server_key: "3b348bf4d4bdae47ed196fe95ccb70c0"
server_key_iv: "5bc4b69d8453fd809ba3723bc52d589b"
server_key_version: "encryptionkey@0"
I20260619 06:58:17.525985 15760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:17.538589 15760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:17.541765 15760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:17.541992 15760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:17.549748 15760 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:45101
I20260619 06:58:17.550135 15826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:45101 every 8 connection(s)
I20260619 06:58:17.550513 15760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:17.553243 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:17.558220 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 15760
I20260619 06:58:17.558389 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:17.558668 11689 external_mini_cluster.cc:1428] Setting key 111ea1defe97846dc73345c376e15aea
I20260619 06:58:17.558904 15827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094: Bootstrap starting.
I20260619 06:58:17.561628 15827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:17.562551 15827 log.cc:824] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:17.564436 15827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094: No bootstrap required, opened a new log
Jun 19 06:58:17 dist-test-slave-10d0 krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:17.568265 15827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911fb843aaf48b4a0e9d1d24ac53094" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45101 } }
I20260619 06:58:17.568519 15827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:17.568622 15827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0911fb843aaf48b4a0e9d1d24ac53094, State: Initialized, Role: FOLLOWER
I20260619 06:58:17.569119 15827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911fb843aaf48b4a0e9d1d24ac53094" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45101 } }
I20260619 06:58:17.569278 15827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:17.569358 15827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:17.569478 15827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:17.570559 15827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911fb843aaf48b4a0e9d1d24ac53094" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45101 } }
I20260619 06:58:17.570923 15827 leader_election.cc:302] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0911fb843aaf48b4a0e9d1d24ac53094; no voters:
I20260619 06:58:17.571269 15827 leader_election.cc:288] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:17.571341 15832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:17.571538 15832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [term 1 LEADER]: Becoming Leader. State: Replica: 0911fb843aaf48b4a0e9d1d24ac53094, State: Running, Role: LEADER
I20260619 06:58:17.571849 15832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911fb843aaf48b4a0e9d1d24ac53094" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45101 } }
I20260619 06:58:17.572619 15827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:17.573480 15833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0911fb843aaf48b4a0e9d1d24ac53094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911fb843aaf48b4a0e9d1d24ac53094" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45101 } } }
I20260619 06:58:17.573635 15833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:17.573971 15834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0911fb843aaf48b4a0e9d1d24ac53094. Latest consensus state: current_term: 1 leader_uuid: "0911fb843aaf48b4a0e9d1d24ac53094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911fb843aaf48b4a0e9d1d24ac53094" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45101 } } }
I20260619 06:58:17.574057 15834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:17.575167 15841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:17.575202 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:17.560168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.11.106.126:45101)
0619 06:58:17.560630 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:17.560650 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:17.560694 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:58:17.561540 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:17.561568 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:17.562034 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:17.562398 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:17.563244 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:17.564298 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:17.565356 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:17.565882 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:17.566071 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:17.569027 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:17.569033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:17.569048 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:17.569077 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:17.571239 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:17.571896 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:17.571902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:17.571908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:17.572064 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:17.572404 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:17.572408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:17.572412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:17.573032 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:17.573232 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:17.573766 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 06:58:17.574105 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260619 06:58:17.578523 15841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:17.584688 15841 catalog_manager.cc:1347] Generated new cluster ID: 2b0bf642dc764d818ad3110a329911ba
I20260619 06:58:17.584772 15841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:17.599004 15841 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:17.600950 15841 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:17.610071 15841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094: Generated new TSK 0
I20260619 06:58:17.610836 15841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:58:17.644348 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45101
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:44063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:17.767722 15853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:17.768008 15853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:17.768055 15853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:17.772341 15853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:17.772413 15853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:17.772477 15853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:17.778256 15853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:44063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:17.779376 15853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:17.780380 15853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:17.790401 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:17.790391 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:17.790391 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:17.790709 15853 server_base.cc:1034] running on GCE node
I20260619 06:58:17.791096 15853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:17.791736 15853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:17.792936 15853 hybrid_clock.cc:648] HybridClock initialized: now 1781852297792922 us; error 27 us; skew 500 ppm
Jun 19 06:58:17 dist-test-slave-10d0 krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:17.795928 15853 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:58:17.796682 15853 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:17.796752 15853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:17.796927 15853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:17.798990 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "b6b5d5cb05484ebfbdc45e20c8069207"
format_stamp: "Formatted at 2026-06-19 06:58:17 on dist-test-slave-10d0"
server_key: "009bd0e956cb5b110eff621f79b78450"
server_key_iv: "9e5b8bdff7ef84f8a38b7a993363dedd"
server_key_version: "encryptionkey@0"
I20260619 06:58:17.799500 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "b6b5d5cb05484ebfbdc45e20c8069207"
format_stamp: "Formatted at 2026-06-19 06:58:17 on dist-test-slave-10d0"
server_key: "009bd0e956cb5b110eff621f79b78450"
server_key_iv: "9e5b8bdff7ef84f8a38b7a993363dedd"
server_key_version: "encryptionkey@0"
I20260619 06:58:17.803262 15853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:58:17.805868 15867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:17.807193 15853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:58:17.807303 15853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "b6b5d5cb05484ebfbdc45e20c8069207"
format_stamp: "Formatted at 2026-06-19 06:58:17 on dist-test-slave-10d0"
server_key: "009bd0e956cb5b110eff621f79b78450"
server_key_iv: "9e5b8bdff7ef84f8a38b7a993363dedd"
server_key_version: "encryptionkey@0"
I20260619 06:58:17.807394 15853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:17.829391 15853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:17.832805 15853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:17.832964 15853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:17.833514 15853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:17.834334 15853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:17.834380 15853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:17.834420 15853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:17.834436 15853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:17.843708 15853 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:45997
I20260619 06:58:17.843741 15980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:45997 every 8 connection(s)
I20260619 06:58:17.844353 15853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:17.852486 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 15853
I20260619 06:58:17.852682 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:17.853060 11689 external_mini_cluster.cc:1428] Setting key 2ab1fac37ce1713b24d54835539dae7a
Jun 19 06:58:17 dist-test-slave-10d0 krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852297, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:17.858721 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:17.846253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:39795 (local address 127.11.106.126:45101)
0619 06:58:17.846397 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:17.846401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:17.847282 (+ 881us) server_negotiation.cc:407] Connection header received
0619 06:58:17.848637 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:17.848640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:17.848701 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:17.848787 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:17.849961 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:17.850500 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:17.851667 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:17.851918 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:17.852037 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:17.854810 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:17.854813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:17.854816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:17.854839 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:17.856293 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:17.856988 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:17.856991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:17.856993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:17.857099 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:17.857537 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:17.857540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:17.857541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:17.857717 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:17.857793 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:17.858484 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 06:58:17.858597 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:58:17.860018 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:17.846595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45101 (local address 127.11.106.65:39795)
0619 06:58:17.847107 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:17.847156 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:17.848397 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:17.848960 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:17.848968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:17.849330 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:17.849806 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:17.849820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:17.850632 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:17.850637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:17.851535 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:17.851545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:17.852081 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:17.852085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:17.852315 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:17.853000 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:17.853022 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:17.854610 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:17.856415 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:17.856423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:17.856441 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:17.856838 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:17.857221 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:17.857226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:17.857229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:17.857430 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:17.857828 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:17.857837 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:17.858179 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 06:58:17.858418 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 06:58:17.858817 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260619 06:58:17.861388 15981 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45101
I20260619 06:58:17.861637 15981 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:17.862111 15981 heartbeater.cc:511] Master 127.11.106.126:45101 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 06:58:17.863644 15791 ts_manager.cc:194] Registered new tserver with Master: b6b5d5cb05484ebfbdc45e20c8069207 (127.11.106.65:45997)
I20260619 06:58:17.864959 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:39795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:58:17.880334 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45101
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:44063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:17.995014 15986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:17.995282 15986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:17.995327 15986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:17.999657 15986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:17.999724 15986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:17.999816 15986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:18.005291 15986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:44063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:18.006484 15986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:18.007555 15986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:18.017581 15992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:18.017584 15991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:18.017688 15986 server_base.cc:1034] running on GCE node
W20260619 06:58:18.017581 15994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:18.018355 15986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:18.019027 15986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:18.020237 15986 hybrid_clock.cc:648] HybridClock initialized: now 1781852298020197 us; error 53 us; skew 500 ppm
Jun 19 06:58:18 dist-test-slave-10d0 krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:18.023384 15986 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:58:18.024279 15986 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:18.024379 15986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:18.024664 15986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:18.026859 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "013e44cd92944489bcafa2fc34152011"
format_stamp: "Formatted at 2026-06-19 06:58:18 on dist-test-slave-10d0"
server_key: "05d8a27d69d4b4c65aa3096dd0076272"
server_key_iv: "a9b88278547f00be1c70c26502e7a4ed"
server_key_version: "encryptionkey@0"
I20260619 06:58:18.027464 15986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "013e44cd92944489bcafa2fc34152011"
format_stamp: "Formatted at 2026-06-19 06:58:18 on dist-test-slave-10d0"
server_key: "05d8a27d69d4b4c65aa3096dd0076272"
server_key_iv: "a9b88278547f00be1c70c26502e7a4ed"
server_key_version: "encryptionkey@0"
I20260619 06:58:18.031673 15986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 06:58:18.034531 16000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:18.035939 15986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:18.036115 15986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "013e44cd92944489bcafa2fc34152011"
format_stamp: "Formatted at 2026-06-19 06:58:18 on dist-test-slave-10d0"
server_key: "05d8a27d69d4b4c65aa3096dd0076272"
server_key_iv: "a9b88278547f00be1c70c26502e7a4ed"
server_key_version: "encryptionkey@0"
I20260619 06:58:18.036243 15986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:18.054150 15986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:18.057412 15986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:18.057619 15986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:18.058274 15986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:18.059154 15986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:18.059237 15986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:18.059307 15986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:18.059357 15986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:18.068843 15986 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:33177
I20260619 06:58:18.068861 16113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:33177 every 8 connection(s)
I20260619 06:58:18.069469 15986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:18.077374 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 15986
I20260619 06:58:18.077461 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:18.077780 11689 external_mini_cluster.cc:1428] Setting key 2ff2885743fe9eec70892347fa2d4858
Jun 19 06:58:18 dist-test-slave-10d0 krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:18.084151 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.071192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:40967 (local address 127.11.106.126:45101)
0619 06:58:18.071365 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.071370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.072297 (+ 927us) server_negotiation.cc:407] Connection header received
0619 06:58:18.073814 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.073818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.073872 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.073994 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:18.075420 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.075965 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.077187 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.077412 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.077539 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.080449 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:18.080456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:18.080462 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:18.080495 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:18.082006 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:18.082516 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:18.082518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:18.082520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:18.082590 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:18.082980 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:18.082982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:18.082983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:18.083148 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:18.083228 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.083861 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.083958 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:58:18.085366 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.071630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45101 (local address 127.11.106.66:40967)
0619 06:58:18.072084 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:18.072184 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:18.073506 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:18.074208 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:18.074219 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:18.074772 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:18.075261 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.075285 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.076099 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.076102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.077059 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.077068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.077646 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.077650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.077867 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.078557 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:18.078579 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:18.080279 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:18.082123 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:18.082130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:18.082146 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:18.082427 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:18.082678 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:18.082684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:18.082688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:18.082881 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:18.083253 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:18.083263 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:18.083594 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 06:58:18.083823 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 06:58:18.084196 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260619 06:58:18.086674 16114 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45101
I20260619 06:58:18.086939 16114 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:18.087415 16114 heartbeater.cc:511] Master 127.11.106.126:45101 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 06:58:18.088531 15791 ts_manager.cc:194] Registered new tserver with Master: 013e44cd92944489bcafa2fc34152011 (127.11.106.66:33177)
I20260619 06:58:18.089145 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:40967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:58:18.108268 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45101
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:44063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:18.226891 16119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:18.227165 16119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:18.227247 16119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:18.231496 16119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:18.231557 16119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:18.231621 16119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:18.237147 16119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:44063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:18.238265 16119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:18.239439 16119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:18.249119 16124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:18.249125 16127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:18.249119 16125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:18.249480 16119 server_base.cc:1034] running on GCE node
I20260619 06:58:18.249979 16119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:18.250692 16119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:18.251921 16119 hybrid_clock.cc:648] HybridClock initialized: now 1781852298251901 us; error 32 us; skew 500 ppm
Jun 19 06:58:18 dist-test-slave-10d0 krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:18.254946 16119 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:58:18.255784 16119 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:18.255879 16119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:18.256120 16119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:18.258380 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3"
format_stamp: "Formatted at 2026-06-19 06:58:18 on dist-test-slave-10d0"
server_key: "a82a4e956ef167f54d75a9f2e720baf2"
server_key_iv: "ce0eeb02358e97c5e3afdfc00c8db8fa"
server_key_version: "encryptionkey@0"
I20260619 06:58:18.258996 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3"
format_stamp: "Formatted at 2026-06-19 06:58:18 on dist-test-slave-10d0"
server_key: "a82a4e956ef167f54d75a9f2e720baf2"
server_key_iv: "ce0eeb02358e97c5e3afdfc00c8db8fa"
server_key_version: "encryptionkey@0"
I20260619 06:58:18.263016 16119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260619 06:58:18.265723 16133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:18.267102 16119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:58:18.267282 16119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3"
format_stamp: "Formatted at 2026-06-19 06:58:18 on dist-test-slave-10d0"
server_key: "a82a4e956ef167f54d75a9f2e720baf2"
server_key_iv: "ce0eeb02358e97c5e3afdfc00c8db8fa"
server_key_version: "encryptionkey@0"
I20260619 06:58:18.267413 16119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:18.286803 16119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:18.290259 16119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:18.290467 16119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:18.291091 16119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:18.291978 16119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:18.292048 16119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:18.292119 16119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:18.292171 16119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:18.301616 16119 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:39817
I20260619 06:58:18.301632 16246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:39817 every 8 connection(s)
I20260619 06:58:18.302313 16119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:18.305831 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 16119
I20260619 06:58:18.305985 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:18.306277 11689 external_mini_cluster.cc:1428] Setting key 820064bf44db4ddf675f83d8cd0a90d8
Jun 19 06:58:18 dist-test-slave-10d0 krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852298, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:18.317500 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.304232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:56267 (local address 127.11.106.126:45101)
0619 06:58:18.304411 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.304415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.305357 (+ 942us) server_negotiation.cc:407] Connection header received
0619 06:58:18.306934 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.306938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.306998 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.307100 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:18.308474 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.309076 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.310340 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.310588 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.310750 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.313565 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:18.313568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:18.313570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:18.313597 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:18.315100 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:18.315695 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:18.315698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:18.315700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:18.315853 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:18.316236 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:18.316239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:18.316240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:18.316437 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:18.316518 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.317244 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.317365 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:58:18.318545 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.304681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45101 (local address 127.11.106.67:56267)
0619 06:58:18.305173 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:18.305222 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:18.306679 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:18.307336 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:18.307348 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:18.307784 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:18.308323 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.308339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.309207 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.309211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.310138 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.310148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.310801 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.310806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.310992 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.311730 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:18.311749 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:18.313388 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:18.315233 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:18.315240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:18.315256 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:18.315579 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:18.315974 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:18.315976 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:18.315978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:18.316147 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:18.316636 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:18.316643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:18.316947 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 06:58:18.317166 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 06:58:18.317501 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260619 06:58:18.320111 16247 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45101
I20260619 06:58:18.320413 16247 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:18.321045 16247 heartbeater.cc:511] Master 127.11.106.126:45101 requested a full tablet report, sending...
I20260619 06:58:18.322212 15790 ts_manager.cc:194] Registered new tserver with Master: 1a7bf06ad7ea4815a8896d7e6ff6caa3 (127.11.106.67:39817)
I20260619 06:58:18.322940 15790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:56267
I20260619 06:58:18.332858 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:58:18.343590 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.335087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.11.106.126:45101)
0619 06:58:18.335359 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.335364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.335532 (+ 168us) server_negotiation.cc:407] Connection header received
0619 06:58:18.335758 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.335761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.335807 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.335895 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:18.336698 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.337249 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.338136 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.338375 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.338484 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.340062 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:18.340067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:18.340070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:18.340108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:18.341776 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:18.342258 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:18.342263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:18.342265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:18.342355 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:18.342638 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:18.342642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:18.342644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:18.342895 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:18.343040 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.343289 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.343413 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:58:18.347038 15790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:18.350447 15790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:18.367166 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.360273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:33177 (local address 127.0.0.1:35300)
0619 06:58:18.361544 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:18.361561 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:18.361650 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:18.362471 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:18.362478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:18.362517 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:18.362879 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.362885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.364385 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.364388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.365419 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.365428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.366496 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.366499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.366588 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.366622 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:58:18.366830 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 06:58:18.366977 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"mutex_wait_us":250,"spinlock_wait_cycles":7040,"thread_start_us":124,"threads_started":1}
I20260619 06:58:18.367163 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.361632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.11.106.66:33177)
0619 06:58:18.361985 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.361998 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.362017 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:58:18.362080 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.362084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.362275 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.362451 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:18.363050 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.364238 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.365553 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.366352 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.366470 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.366663 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.366879 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.366994 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":123,"threads_started":1}
I20260619 06:58:18.368319 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.360175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:39817 (local address 127.0.0.1:42014)
0619 06:58:18.360824 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:18.360884 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:18.361030 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:18.362477 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:18.362479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:18.362515 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:18.362862 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.362872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.364134 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.364137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.365797 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.365808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.367983 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.367986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.368063 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.368082 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:58:18.368132 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 06:58:18.368205 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":203,"spinlock_wait_cycles":13184,"thread_start_us":110,"threads_started":1}
I20260619 06:58:18.368446 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.361027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.11.106.67:39817)
0619 06:58:18.362067 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.362078 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.362098 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:58:18.362165 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.362169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.362315 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.362662 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:18.362998 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.363986 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.367148 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.367847 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.367997 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.368159 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.368210 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.368302 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":202,"threads_started":1}
I20260619 06:58:18.369129 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.360572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45997 (local address 127.0.0.1:32852)
0619 06:58:18.361811 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:18.361824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:18.361928 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:18.362730 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:18.362732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:18.362747 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:18.362918 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.362924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.364430 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.364433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.365797 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.365808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.368684 (+ 2876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.368687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.368785 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.368811 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:58:18.368893 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:58:18.368991 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"mutex_wait_us":114,"thread_start_us":98,"threads_started":1}
I20260619 06:58:18.369400 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.361160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.11.106.65:45997)
0619 06:58:18.361670 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.361678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.361934 (+ 256us) server_negotiation.cc:407] Connection header received
0619 06:58:18.362038 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.362046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.362276 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.362450 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:18.363069 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.364266 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.365940 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.366826 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.366935 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.367073 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.368952 (+ 1879us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.369063 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":102,"threads_started":1}
I20260619 06:58:18.371359 16048 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2cd2f48e934174b552f7151facae42 (DEFAULT_TABLE table=test-table [id=090fcbdaf4234606bf13c6b2a0c7108b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:18.371544 16181 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2cd2f48e934174b552f7151facae42 (DEFAULT_TABLE table=test-table [id=090fcbdaf4234606bf13c6b2a0c7108b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:18.372219 15915 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2cd2f48e934174b552f7151facae42 (DEFAULT_TABLE table=test-table [id=090fcbdaf4234606bf13c6b2a0c7108b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:18.372749 16181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2cd2f48e934174b552f7151facae42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:18.372994 16048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2cd2f48e934174b552f7151facae42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:18.373458 15915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2cd2f48e934174b552f7151facae42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:18.380648 16264 tablet_bootstrap.cc:492] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: Bootstrap starting.
I20260619 06:58:18.382266 16265 tablet_bootstrap.cc:492] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207: Bootstrap starting.
I20260619 06:58:18.383654 16264 tablet_bootstrap.cc:654] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:18.384558 16265 tablet_bootstrap.cc:654] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:18.385092 16264 log.cc:824] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:18.385421 16265 log.cc:824] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:18.385937 16266 tablet_bootstrap.cc:492] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: Bootstrap starting.
I20260619 06:58:18.387377 16265 tablet_bootstrap.cc:492] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207: No bootstrap required, opened a new log
I20260619 06:58:18.387455 16264 tablet_bootstrap.cc:492] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: No bootstrap required, opened a new log
I20260619 06:58:18.387636 16265 ts_tablet_manager.cc:1397] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:58:18.387705 16264 ts_tablet_manager.cc:1397] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:58:18.388193 16266 tablet_bootstrap.cc:654] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:18.389577 16266 log.cc:824] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:18.391811 16265 raft_consensus.cc:348] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } }
I20260619 06:58:18.391832 16264 raft_consensus.cc:348] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } }
I20260619 06:58:18.392081 16265 raft_consensus.cc:374] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:18.392131 16265 raft_consensus.cc:729] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b5d5cb05484ebfbdc45e20c8069207, State: Initialized, Role: FOLLOWER
I20260619 06:58:18.392174 16264 raft_consensus.cc:374] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:18.392251 16264 raft_consensus.cc:729] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013e44cd92944489bcafa2fc34152011, State: Initialized, Role: FOLLOWER
I20260619 06:58:18.392724 16265 consensus_queue.cc:260] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } }
I20260619 06:58:18.392766 16264 consensus_queue.cc:260] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } }
I20260619 06:58:18.393455 15981 heartbeater.cc:503] Master 127.11.106.126:45101 was elected leader, sending a full tablet report...
I20260619 06:58:18.393778 16265 ts_tablet_manager.cc:1428] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 06:58:18.393797 16264 ts_tablet_manager.cc:1428] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 06:58:18.394145 16114 heartbeater.cc:503] Master 127.11.106.126:45101 was elected leader, sending a full tablet report...
I20260619 06:58:18.395128 16266 tablet_bootstrap.cc:492] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: No bootstrap required, opened a new log
I20260619 06:58:18.395394 16266 ts_tablet_manager.cc:1397] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260619 06:58:18.399080 16266 raft_consensus.cc:348] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } }
I20260619 06:58:18.399439 16266 raft_consensus.cc:374] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:18.399507 16266 raft_consensus.cc:729] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a7bf06ad7ea4815a8896d7e6ff6caa3, State: Initialized, Role: FOLLOWER
I20260619 06:58:18.399974 16266 consensus_queue.cc:260] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } }
I20260619 06:58:18.400676 16247 heartbeater.cc:503] Master 127.11.106.126:45101 was elected leader, sending a full tablet report...
I20260619 06:58:18.401006 16266 ts_tablet_manager.cc:1428] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 06:58:18.553241 16248 tablet.cc:2369] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:58:18.570559 16115 tablet.cc:2369] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:18.577527 16271 raft_consensus.cc:482] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:18.577785 16271 raft_consensus.cc:504] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } }
I20260619 06:58:18.579031 16271 leader_election.cc:288] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a7bf06ad7ea4815a8896d7e6ff6caa3 (127.11.106.67:39817), b6b5d5cb05484ebfbdc45e20c8069207 (127.11.106.65:45997)
I20260619 06:58:18.583832 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.579460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:34547 (local address 127.11.106.65:45997)
0619 06:58:18.579644 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.579649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.579695 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:58:18.579763 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.579765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.579806 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.579867 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:18.580226 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.580952 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.582509 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.583147 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.583298 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.583365 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.583615 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.583703 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 06:58:18.583837 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.579360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45997 (local address 127.11.106.66:34547)
0619 06:58:18.579493 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:18.579511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:18.579623 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:18.579916 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:18.579922 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:18.579946 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:18.580111 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.580117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.581076 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.581079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.582365 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.582377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.583309 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.583312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.583457 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.583486 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:58:18.583581 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:58:18.583689 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":5}
I20260619 06:58:18.583876 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.579485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:60411 (local address 127.11.106.67:39817)
0619 06:58:18.579641 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.579647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.579971 (+ 324us) server_negotiation.cc:407] Connection header received
0619 06:58:18.580079 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.580081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.580119 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.580211 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:18.580588 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.581225 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.582707 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.583306 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.583421 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.583455 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.583660 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.583771 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:58:18.583850 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.579371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:39817 (local address 127.11.106.66:60411)
0619 06:58:18.579831 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:18.579867 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:18.579960 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:18.580233 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:18.580236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:18.580263 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:18.580414 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.580425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.581344 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.581348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.582536 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:18.582544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.583430 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.583433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:18.583526 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.583556 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:58:18.583625 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 06:58:18.583708 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":147,"thread_start_us":88,"threads_started":1}
I20260619 06:58:18.584659 15935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2cd2f48e934174b552f7151facae42" candidate_uuid: "013e44cd92944489bcafa2fc34152011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" is_pre_election: true
I20260619 06:58:18.584657 16201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2cd2f48e934174b552f7151facae42" candidate_uuid: "013e44cd92944489bcafa2fc34152011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" is_pre_election: true
I20260619 06:58:18.584990 15935 raft_consensus.cc:2454] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 013e44cd92944489bcafa2fc34152011 in term 0.
I20260619 06:58:18.584990 16201 raft_consensus.cc:2454] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 013e44cd92944489bcafa2fc34152011 in term 0.
I20260619 06:58:18.585520 16002 leader_election.cc:302] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013e44cd92944489bcafa2fc34152011, b6b5d5cb05484ebfbdc45e20c8069207; no voters:
I20260619 06:58:18.585776 16271 raft_consensus.cc:2781] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:18.585861 16271 raft_consensus.cc:482] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:18.585906 16271 raft_consensus.cc:3037] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:18.586917 16271 raft_consensus.cc:504] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } }
I20260619 06:58:18.587277 16271 leader_election.cc:288] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [CANDIDATE]: Term 1 election: Requested vote from peers 1a7bf06ad7ea4815a8896d7e6ff6caa3 (127.11.106.67:39817), b6b5d5cb05484ebfbdc45e20c8069207 (127.11.106.65:45997)
I20260619 06:58:18.587646 15935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2cd2f48e934174b552f7151facae42" candidate_uuid: "013e44cd92944489bcafa2fc34152011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6b5d5cb05484ebfbdc45e20c8069207"
I20260619 06:58:18.587625 16201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2cd2f48e934174b552f7151facae42" candidate_uuid: "013e44cd92944489bcafa2fc34152011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3"
I20260619 06:58:18.587765 15935 raft_consensus.cc:3037] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:18.587765 16201 raft_consensus.cc:3037] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:18.589092 16201 raft_consensus.cc:2454] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 013e44cd92944489bcafa2fc34152011 in term 1.
I20260619 06:58:18.589092 15935 raft_consensus.cc:2454] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 013e44cd92944489bcafa2fc34152011 in term 1.
I20260619 06:58:18.589434 16002 leader_election.cc:302] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013e44cd92944489bcafa2fc34152011, b6b5d5cb05484ebfbdc45e20c8069207; no voters:
I20260619 06:58:18.589639 16271 raft_consensus.cc:2781] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:18.589865 16271 raft_consensus.cc:686] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 1 LEADER]: Becoming Leader. State: Replica: 013e44cd92944489bcafa2fc34152011, State: Running, Role: LEADER
I20260619 06:58:18.590219 16271 consensus_queue.cc:237] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } }
I20260619 06:58:18.593513 15790 catalog_manager.cc:5495] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 reported cstate change: term changed from 0 to 1, leader changed from <none> to 013e44cd92944489bcafa2fc34152011 (127.11.106.66). New cstate: current_term: 1 leader_uuid: "013e44cd92944489bcafa2fc34152011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "013e44cd92944489bcafa2fc34152011" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 } health_report { overall_health: UNKNOWN } } }
W20260619 06:58:18.595727 15982 tablet.cc:2369] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:18.662571 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.658647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.11.106.66:33177)
0619 06:58:18.658803 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.658807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.658823 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:18.658954 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.658956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.659015 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.659115 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:18.659401 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.659937 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.660956 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:18.661177 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:18.661266 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:18.661608 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:18.662086 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:18.662188 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.662366 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.662434 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:58:18.669355 15935 raft_consensus.cc:1261] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207 [term 1 FOLLOWER]: Refusing update from remote peer 013e44cd92944489bcafa2fc34152011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:18.669355 16201 raft_consensus.cc:1261] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [term 1 FOLLOWER]: Refusing update from remote peer 013e44cd92944489bcafa2fc34152011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:18.670135 16271 consensus_queue.cc:1035] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6b5d5cb05484ebfbdc45e20c8069207" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:18.670349 16274 consensus_queue.cc:1035] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a7bf06ad7ea4815a8896d7e6ff6caa3" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:18.686710 15790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:18.686921 15790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:18.689450 15790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0911fb843aaf48b4a0e9d1d24ac53094: Sending DeleteTablet for 3 replicas of tablet 6c2cd2f48e934174b552f7151facae42
I20260619 06:58:18.690410 15915 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c2cd2f48e934174b552f7151facae42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:18 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:32852
I20260619 06:58:18.690404 16181 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c2cd2f48e934174b552f7151facae42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:18 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:42014
I20260619 06:58:18.690975 16286 tablet_replica.cc:330] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207: stopping tablet replica
I20260619 06:58:18.690979 16048 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c2cd2f48e934174b552f7151facae42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:18 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:35300
I20260619 06:58:18.691447 16286 raft_consensus.cc:2229] T 6c2cd2f48e934174b552f7151facae42 P b6b5d5cb05484ebfbdc45e20c8069207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:18.691650 16287 tablet_replica.cc:330] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: stopping tablet replica
I20260619 06:58:18.691782 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 15853
I20260619 06:58:18.691973 16287 raft_consensus.cc:2229] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:18.692286 16287 raft_consensus.cc:2258] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:18.694046 16287 ts_tablet_manager.cc:1905] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:18.694195 16288 tablet_replica.cc:330] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: stopping tablet replica
I20260619 06:58:18.694545 16288 raft_consensus.cc:2229] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:18.695065 16288 raft_consensus.cc:2258] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:18.696751 16288 ts_tablet_manager.cc:1905] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:18.696913 16287 ts_tablet_manager.cc:1918] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:18.697012 16287 log.cc:1196] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/6c2cd2f48e934174b552f7151facae42
I20260619 06:58:18.697343 16287 ts_tablet_manager.cc:1939] T 6c2cd2f48e934174b552f7151facae42 P 1a7bf06ad7ea4815a8896d7e6ff6caa3: Deleting consensus metadata
I20260619 06:58:18.698691 15775 catalog_manager.cc:4841] TS 1a7bf06ad7ea4815a8896d7e6ff6caa3 (127.11.106.67:39817): tablet 6c2cd2f48e934174b552f7151facae42 (table test-table [id=090fcbdaf4234606bf13c6b2a0c7108b]) successfully deleted
I20260619 06:58:18.700510 16288 ts_tablet_manager.cc:1918] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 06:58:18.700569 15776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:45997 (error 108)
I20260619 06:58:18.700641 16288 log.cc:1196] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/6c2cd2f48e934174b552f7151facae42
I20260619 06:58:18.700985 16288 ts_tablet_manager.cc:1939] T 6c2cd2f48e934174b552f7151facae42 P 013e44cd92944489bcafa2fc34152011: Deleting consensus metadata
I20260619 06:58:18.701702 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.701393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45997 (local address 127.0.0.1:32862)
0619 06:58:18.701539 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:18.701611 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:45997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 06:58:18.701747 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 15986
W20260619 06:58:18.703101 15776 catalog_manager.cc:4584] TS b6b5d5cb05484ebfbdc45e20c8069207 (127.11.106.65:45997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c2cd2f48e934174b552f7151facae42: Network error: Client connection negotiation failed: client connection to 127.11.106.65:45997: connect: Connection refused (error 111)
I20260619 06:58:18.708475 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 16119
I20260619 06:58:18.709360 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.708737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:33177 (local address 127.0.0.1:35310)
0619 06:58:18.709166 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:18.709244 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.66:33177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315}
I20260619 06:58:18.714300 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 15760
2026-06-19T06:58:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2515 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T06:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:18Z Disabled control of system clock
I20260619 06:58:18.745268 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42247
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:42247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:58:18.861362 16293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:18.861649 16293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:18.861688 16293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:18.865733 16293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:18.865808 16293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:18.865823 16293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:18.865840 16293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:18.865856 16293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:18.870617 16293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:42247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:18.871701 16293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:18.873210 16293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:18.881698 16298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:18.881690 16301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:18.881698 16299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:18.881933 16293 server_base.cc:1034] running on GCE node
I20260619 06:58:18.882536 16293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:18.883690 16293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:18.884879 16293 hybrid_clock.cc:648] HybridClock initialized: now 1781852298884859 us; error 32 us; skew 500 ppm
I20260619 06:58:18.886685 16293 webserver.cc:460] Webserver started at http://127.11.106.126:44835/ using document root <none> and password file <none>
I20260619 06:58:18.887279 16293 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:18.887372 16293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:18.887626 16293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:18.889816 16293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe"
format_stamp: "Formatted at 2026-06-19 06:58:18 on dist-test-slave-10d0"
server_key: "099437f4df3d1607661c4a7cf8096825"
server_key_iv: "e7bed1dcf5e7b1bc5500b68847752eea"
server_key_version: "encryptionkey@0"
I20260619 06:58:18.890398 16293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe"
format_stamp: "Formatted at 2026-06-19 06:58:18 on dist-test-slave-10d0"
server_key: "099437f4df3d1607661c4a7cf8096825"
server_key_iv: "e7bed1dcf5e7b1bc5500b68847752eea"
server_key_version: "encryptionkey@0"
I20260619 06:58:18.894187 16293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:18.896641 16307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:18.897873 16293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:18.898012 16293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe"
format_stamp: "Formatted at 2026-06-19 06:58:18 on dist-test-slave-10d0"
server_key: "099437f4df3d1607661c4a7cf8096825"
server_key_iv: "e7bed1dcf5e7b1bc5500b68847752eea"
server_key_version: "encryptionkey@0"
I20260619 06:58:18.898130 16293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:18.904057 16293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:18.904778 16293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:18.904969 16293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:18.911870 16293 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:42247
I20260619 06:58:18.911892 16359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:42247 every 8 connection(s)
I20260619 06:58:18.913224 16293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:18.916687 16360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:18.921231 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 16293
I20260619 06:58:18.921356 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:18.921636 11689 external_mini_cluster.cc:1428] Setting key 23be1ddef5173c2d4c366056d223420f
I20260619 06:58:18.923048 16360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe: Bootstrap starting.
I20260619 06:58:18.925733 16360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:18.926738 16360 log.cc:824] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:18.927627 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:18.923212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.11.106.126:42247)
0619 06:58:18.923880 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:18.923897 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:18.923935 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:58:18.924662 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:18.924702 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:18.925117 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:18.925407 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:18.926284 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:18.926289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:18.926301 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:18.926311 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:18.926365 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:18.926382 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:18.926469 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:18.926621 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 06:58:18.926925 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":146,"threads_started":1}
I20260619 06:58:18.928792 16360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe: No bootstrap required, opened a new log
I20260619 06:58:18.932343 16360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42247 } }
I20260619 06:58:18.932653 16360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:18.932722 16360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6b2e35cbe34e54ab53a4da19d80dbe, State: Initialized, Role: FOLLOWER
I20260619 06:58:18.933197 16360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42247 } }
I20260619 06:58:18.933351 16360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:18.933439 16360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:18.933537 16360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:18.934437 16360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42247 } }
I20260619 06:58:18.934777 16360 leader_election.cc:302] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c6b2e35cbe34e54ab53a4da19d80dbe; no voters:
I20260619 06:58:18.935135 16360 leader_election.cc:288] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:18.935231 16365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:18.935489 16365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [term 1 LEADER]: Becoming Leader. State: Replica: 0c6b2e35cbe34e54ab53a4da19d80dbe, State: Running, Role: LEADER
I20260619 06:58:18.935802 16365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42247 } }
I20260619 06:58:18.936399 16360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:18.937502 16366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42247 } } }
I20260619 06:58:18.937659 16366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:18.937502 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c6b2e35cbe34e54ab53a4da19d80dbe. Latest consensus state: current_term: 1 leader_uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b2e35cbe34e54ab53a4da19d80dbe" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42247 } } }
I20260619 06:58:18.937922 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:18.938122 16374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:18.941119 16374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:18.947280 16374 catalog_manager.cc:1347] Generated new cluster ID: 52071702ddfd44c19319f7095d398011
I20260619 06:58:18.947351 16374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:18.963073 16374 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:18.963428 16374 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:18.968027 16374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe: Generated new TSK 0
I20260619 06:58:18.968842 16374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:58:18.975113 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42247
--builtin_ntp_servers=127.11.106.84:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:58:19.104674 16384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:19.104967 16384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:19.105010 16384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:19.109126 16384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:19.109190 16384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:19.109249 16384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:19.114180 16384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:19.115293 16384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:19.116724 16384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:19.126585 16392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:19.126606 16389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:19.126597 16390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:19.127247 16384 server_base.cc:1034] running on GCE node
I20260619 06:58:19.127617 16384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:19.128212 16384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:19.129413 16384 hybrid_clock.cc:648] HybridClock initialized: now 1781852299129359 us; error 69 us; skew 500 ppm
I20260619 06:58:19.131309 16384 webserver.cc:460] Webserver started at http://127.11.106.65:44105/ using document root <none> and password file <none>
I20260619 06:58:19.131956 16384 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:19.132051 16384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:19.132288 16384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:19.134464 16384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "67e7c1eca6e84ed6af1d65c206717cda"
format_stamp: "Formatted at 2026-06-19 06:58:19 on dist-test-slave-10d0"
server_key: "56c08140bd35dd9496b62a488829f37e"
server_key_iv: "c0a0326a15ca52cce278bebec1944ba9"
server_key_version: "encryptionkey@0"
I20260619 06:58:19.135042 16384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "67e7c1eca6e84ed6af1d65c206717cda"
format_stamp: "Formatted at 2026-06-19 06:58:19 on dist-test-slave-10d0"
server_key: "56c08140bd35dd9496b62a488829f37e"
server_key_iv: "c0a0326a15ca52cce278bebec1944ba9"
server_key_version: "encryptionkey@0"
I20260619 06:58:19.139688 16384 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 06:58:19.142529 16398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:19.143850 16384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:19.143988 16384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "67e7c1eca6e84ed6af1d65c206717cda"
format_stamp: "Formatted at 2026-06-19 06:58:19 on dist-test-slave-10d0"
server_key: "56c08140bd35dd9496b62a488829f37e"
server_key_iv: "c0a0326a15ca52cce278bebec1944ba9"
server_key_version: "encryptionkey@0"
I20260619 06:58:19.144107 16384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:19.148846 16384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:19.149507 16384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:19.149696 16384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:19.150249 16384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:19.151170 16384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:19.151238 16384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:19.151333 16384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:19.151383 16384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:19.160651 16384 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:36801
I20260619 06:58:19.160725 16511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:36801 every 8 connection(s)
I20260619 06:58:19.161641 16384 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:19.168463 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.163788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:53815 (local address 127.11.106.126:42247)
0619 06:58:19.163984 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.163989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.165263 (+ 1274us) server_negotiation.cc:407] Connection header received
0619 06:58:19.166676 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.166679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.166720 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.166806 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.168070 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.168073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.168075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.168084 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.168101 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.168108 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.168183 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.168317 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.168355 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 06:58:19.169498 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.164243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42247 (local address 127.11.106.65:53815)
0619 06:58:19.165090 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:19.165136 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:19.166439 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:19.166982 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:19.166989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:19.167087 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:19.167804 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:19.167806 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:19.167850 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:19.167927 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:19.168188 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:19.168195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:19.168200 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:58:19.168349 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:58:19.168631 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":106,"threads_started":1}
I20260619 06:58:19.170770 16512 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42247
I20260619 06:58:19.171031 16512 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:19.171500 16512 heartbeater.cc:511] Master 127.11.106.126:42247 requested a full tablet report, sending...
I20260619 06:58:19.171797 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 16384
I20260619 06:58:19.171909 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:19.172200 11689 external_mini_cluster.cc:1428] Setting key 7ceaab6a971ff7bebc9c0062a203d954
I20260619 06:58:19.173367 16324 ts_manager.cc:194] Registered new tserver with Master: 67e7c1eca6e84ed6af1d65c206717cda (127.11.106.65:36801)
I20260619 06:58:19.174266 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42247
--builtin_ntp_servers=127.11.106.84:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:58:19.286021 16515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:19.286326 16515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:19.286401 16515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:19.290385 16515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:19.290469 16515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:19.290561 16515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:19.295207 16515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:19.296300 16515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:19.297720 16515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:19.308082 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:19.308208 16515 server_base.cc:1034] running on GCE node
W20260619 06:58:19.308053 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:19.308014 16520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:19.308728 16515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:19.309396 16515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:19.310604 16515 hybrid_clock.cc:648] HybridClock initialized: now 1781852299310569 us; error 53 us; skew 500 ppm
I20260619 06:58:19.312602 16515 webserver.cc:460] Webserver started at http://127.11.106.66:42171/ using document root <none> and password file <none>
I20260619 06:58:19.313228 16515 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:19.313328 16515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:19.313568 16515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:19.315687 16515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc"
format_stamp: "Formatted at 2026-06-19 06:58:19 on dist-test-slave-10d0"
server_key: "9e36a65214a936939ef9aa5da28fa6fe"
server_key_iv: "b54cfcc880c11a39b140b13a9ca658ae"
server_key_version: "encryptionkey@0"
I20260619 06:58:19.316263 16515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc"
format_stamp: "Formatted at 2026-06-19 06:58:19 on dist-test-slave-10d0"
server_key: "9e36a65214a936939ef9aa5da28fa6fe"
server_key_iv: "b54cfcc880c11a39b140b13a9ca658ae"
server_key_version: "encryptionkey@0"
I20260619 06:58:19.320036 16515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:58:19.322638 16529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:19.323980 16515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:58:19.324118 16515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc"
format_stamp: "Formatted at 2026-06-19 06:58:19 on dist-test-slave-10d0"
server_key: "9e36a65214a936939ef9aa5da28fa6fe"
server_key_iv: "b54cfcc880c11a39b140b13a9ca658ae"
server_key_version: "encryptionkey@0"
I20260619 06:58:19.324244 16515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:19.329021 16515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:19.329638 16515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:19.329823 16515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:19.330361 16515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:19.331271 16515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:19.331316 16515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:19.331384 16515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:19.331423 16515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:19.340679 16515 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:33189
I20260619 06:58:19.340736 16642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:33189 every 8 connection(s)
I20260619 06:58:19.341598 16515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:19.347898 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.343482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:60023 (local address 127.11.106.126:42247)
0619 06:58:19.343639 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.343642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.344808 (+ 1166us) server_negotiation.cc:407] Connection header received
0619 06:58:19.346080 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.346082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.346118 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.346196 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.347424 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.347433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.347435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.347446 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.347466 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.347474 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.347575 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.347718 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.347761 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:58:19.348874 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.343917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42247 (local address 127.11.106.66:60023)
0619 06:58:19.344638 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:19.344701 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:19.345897 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:19.346399 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:19.346406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:19.346493 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:19.347132 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:19.347133 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:19.347175 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:19.347245 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:19.347558 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:19.347567 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:19.347573 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:19.347762 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 06:58:19.348018 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260619 06:58:19.349967 16643 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42247
I20260619 06:58:19.350198 16643 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:19.350590 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 16515
I20260619 06:58:19.350628 16643 heartbeater.cc:511] Master 127.11.106.126:42247 requested a full tablet report, sending...
I20260619 06:58:19.350725 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:19.351006 11689 external_mini_cluster.cc:1428] Setting key b41c8c783e831cb9b4d3807788a58cd4
I20260619 06:58:19.351748 16324 ts_manager.cc:194] Registered new tserver with Master: f1fb4022ac8d4d4d9936e52953cfbfdc (127.11.106.66:33189)
I20260619 06:58:19.353111 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42247
--builtin_ntp_servers=127.11.106.84:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:58:19.466205 16646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:19.466476 16646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:19.466516 16646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:19.470602 16646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:19.470655 16646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:19.470717 16646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:19.475324 16646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:19.476343 16646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:19.477809 16646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:19.487676 16652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:19.487711 16654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:19.487676 16651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:19.487891 16646 server_base.cc:1034] running on GCE node
I20260619 06:58:19.488530 16646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:19.489264 16646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:19.490428 16646 hybrid_clock.cc:648] HybridClock initialized: now 1781852299490403 us; error 38 us; skew 500 ppm
I20260619 06:58:19.492334 16646 webserver.cc:460] Webserver started at http://127.11.106.67:38857/ using document root <none> and password file <none>
I20260619 06:58:19.493013 16646 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:19.493077 16646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:19.493319 16646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:19.495404 16646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "4f38358518a34a2b8910320239ebd9d5"
format_stamp: "Formatted at 2026-06-19 06:58:19 on dist-test-slave-10d0"
server_key: "b2c1090ba6fb19154f812e0d638b6852"
server_key_iv: "f22e30bc059eae846f8645081cd61c50"
server_key_version: "encryptionkey@0"
I20260619 06:58:19.495977 16646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "4f38358518a34a2b8910320239ebd9d5"
format_stamp: "Formatted at 2026-06-19 06:58:19 on dist-test-slave-10d0"
server_key: "b2c1090ba6fb19154f812e0d638b6852"
server_key_iv: "f22e30bc059eae846f8645081cd61c50"
server_key_version: "encryptionkey@0"
I20260619 06:58:19.499779 16646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:58:19.502360 16660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:19.503860 16646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:19.503985 16646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "4f38358518a34a2b8910320239ebd9d5"
format_stamp: "Formatted at 2026-06-19 06:58:19 on dist-test-slave-10d0"
server_key: "b2c1090ba6fb19154f812e0d638b6852"
server_key_iv: "f22e30bc059eae846f8645081cd61c50"
server_key_version: "encryptionkey@0"
I20260619 06:58:19.504078 16646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:19.508575 16646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:19.509215 16646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:19.509348 16646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:19.509817 16646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:19.510641 16646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:19.510686 16646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:19.510731 16646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:19.510753 16646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:19.519500 16646 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:38731
I20260619 06:58:19.519524 16773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:38731 every 8 connection(s)
I20260619 06:58:19.520426 16646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:19.526857 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.522313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:39867 (local address 127.11.106.126:42247)
0619 06:58:19.522468 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.522472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.523526 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 06:58:19.524958 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.524962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.525014 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.525119 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.526357 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.526361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.526364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.526378 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.526409 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.526418 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.526526 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.526672 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.526720 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:58:19.527774 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.522698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42247 (local address 127.11.106.67:39867)
0619 06:58:19.523373 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:19.523415 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:19.524701 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:19.525298 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:19.525305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:19.525397 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:19.526071 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:19.526074 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:19.526123 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:19.526195 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:19.526507 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:19.526515 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:19.526521 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:19.526708 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 06:58:19.526963 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260619 06:58:19.529014 16774 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42247
I20260619 06:58:19.529270 16774 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:19.529428 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 16646
I20260619 06:58:19.529543 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:19.529759 16774 heartbeater.cc:511] Master 127.11.106.126:42247 requested a full tablet report, sending...
I20260619 06:58:19.529841 11689 external_mini_cluster.cc:1428] Setting key 98eb23218cd1333f65ab042749a14278
I20260619 06:58:19.530925 16323 ts_manager.cc:194] Registered new tserver with Master: 4f38358518a34a2b8910320239ebd9d5 (127.11.106.67:38731)
I20260619 06:58:19.531978 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:58:19.535315 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.533843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.11.106.126:42247)
0619 06:58:19.533993 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.533997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.534174 (+ 177us) server_negotiation.cc:407] Connection header received
0619 06:58:19.534265 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.534279 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.534309 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.534372 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.534958 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.534960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.534962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.534972 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.534988 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.534996 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.535062 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.535124 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.535154 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:58:19.538017 16324 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:19.541030 16324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:19.552407 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.549664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:33189 (local address 127.0.0.1:43378)
0619 06:58:19.549922 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:19.549937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:19.550023 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:19.550916 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:19.550919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:19.550929 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:19.551652 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:19.551655 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:19.551667 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:19.551704 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:19.552041 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:19.552051 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:19.552057 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:19.552159 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 06:58:19.552230 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":109,"threads_started":1}
I20260619 06:58:19.552618 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.549279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:38731 (local address 127.0.0.1:56978)
0619 06:58:19.549632 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:19.549681 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:19.549783 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:19.550791 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:19.550797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:19.550878 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:19.551558 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:19.551562 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:19.551583 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:19.551673 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:19.552303 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:19.552306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:19.552309 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:58:19.552390 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 06:58:19.552425 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":108,"threads_started":1}
I20260619 06:58:19.552877 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.549454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.11.106.67:38731)
0619 06:58:19.550315 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.550327 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.550353 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:58:19.550410 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.550415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.550611 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.551182 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.552104 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.552112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.552125 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.552139 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.552181 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.552207 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.552305 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.552423 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.552719 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":149,"threads_started":1}
I20260619 06:58:19.553184 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.550137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.11.106.66:33189)
0619 06:58:19.550545 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.550555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.550575 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:58:19.550622 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.550627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.550784 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.551183 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.551841 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.551849 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.551861 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.551871 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.551910 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.551934 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.552897 (+ 963us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.552952 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.553029 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":115,"threads_started":1}
I20260619 06:58:19.553274 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.550083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:36801 (local address 127.0.0.1:53988)
0619 06:58:19.550323 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:19.550337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:19.550416 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:19.551726 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:19.551732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:19.551751 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:19.552152 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:19.552154 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:19.552163 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:19.552190 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:19.553062 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:19.553066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:19.553069 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:58:19.553127 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 06:58:19.553154 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":106,"threads_started":1}
I20260619 06:58:19.553828 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.550495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.11.106.65:36801)
0619 06:58:19.550783 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.550794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.550817 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:58:19.550872 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.550878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.551061 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.551998 (+ 937us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.552863 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.552868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.552884 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.552899 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.552938 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.552964 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.553549 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.553598 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.553667 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
I20260619 06:58:19.555958 16577 tablet_service.cc:1467] Processing CreateTablet for tablet c487b80d298048ddbeb36915ae5a367d (DEFAULT_TABLE table=test-table [id=3fd8d6af2f8048ddaeedc8c989bff0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:19.555958 16708 tablet_service.cc:1467] Processing CreateTablet for tablet c487b80d298048ddbeb36915ae5a367d (DEFAULT_TABLE table=test-table [id=3fd8d6af2f8048ddaeedc8c989bff0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:19.556128 16446 tablet_service.cc:1467] Processing CreateTablet for tablet c487b80d298048ddbeb36915ae5a367d (DEFAULT_TABLE table=test-table [id=3fd8d6af2f8048ddaeedc8c989bff0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:19.557201 16577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c487b80d298048ddbeb36915ae5a367d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:19.557201 16708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c487b80d298048ddbeb36915ae5a367d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:19.557269 16446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c487b80d298048ddbeb36915ae5a367d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:19.563680 16791 tablet_bootstrap.cc:492] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda: Bootstrap starting.
I20260619 06:58:19.565047 16793 tablet_bootstrap.cc:492] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: Bootstrap starting.
I20260619 06:58:19.565155 16792 tablet_bootstrap.cc:492] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: Bootstrap starting.
I20260619 06:58:19.566356 16791 tablet_bootstrap.cc:654] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:19.567142 16792 tablet_bootstrap.cc:654] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:19.567137 16793 tablet_bootstrap.cc:654] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:19.567130 16791 log.cc:824] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:19.567962 16793 log.cc:824] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:19.567962 16792 log.cc:824] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:19.569202 16791 tablet_bootstrap.cc:492] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda: No bootstrap required, opened a new log
I20260619 06:58:19.569411 16791 ts_tablet_manager.cc:1397] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:58:19.569721 16793 tablet_bootstrap.cc:492] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: No bootstrap required, opened a new log
I20260619 06:58:19.569909 16793 ts_tablet_manager.cc:1397] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:58:19.569926 16792 tablet_bootstrap.cc:492] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: No bootstrap required, opened a new log
I20260619 06:58:19.570088 16792 ts_tablet_manager.cc:1397] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:58:19.573184 16793 raft_consensus.cc:348] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } }
I20260619 06:58:19.573418 16793 raft_consensus.cc:374] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:19.573498 16793 raft_consensus.cc:729] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f38358518a34a2b8910320239ebd9d5, State: Initialized, Role: FOLLOWER
I20260619 06:58:19.573951 16793 consensus_queue.cc:260] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } }
I20260619 06:58:19.573853 16791 raft_consensus.cc:348] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } }
I20260619 06:58:19.574179 16791 raft_consensus.cc:374] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:19.574242 16791 raft_consensus.cc:729] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e7c1eca6e84ed6af1d65c206717cda, State: Initialized, Role: FOLLOWER
I20260619 06:58:19.574635 16774 heartbeater.cc:503] Master 127.11.106.126:42247 was elected leader, sending a full tablet report...
I20260619 06:58:19.574811 16791 consensus_queue.cc:260] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } }
I20260619 06:58:19.574947 16793 ts_tablet_manager.cc:1428] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:58:19.575413 16792 raft_consensus.cc:348] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } }
I20260619 06:58:19.575570 16512 heartbeater.cc:503] Master 127.11.106.126:42247 was elected leader, sending a full tablet report...
I20260619 06:58:19.575687 16792 raft_consensus.cc:374] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:19.575780 16792 raft_consensus.cc:729] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fb4022ac8d4d4d9936e52953cfbfdc, State: Initialized, Role: FOLLOWER
I20260619 06:58:19.575933 16791 ts_tablet_manager.cc:1428] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 06:58:19.576205 16792 consensus_queue.cc:260] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } }
I20260619 06:58:19.576924 16643 heartbeater.cc:503] Master 127.11.106.126:42247 was elected leader, sending a full tablet report...
I20260619 06:58:19.577226 16792 ts_tablet_manager.cc:1428] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260619 06:58:19.592609 16644 tablet.cc:2369] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:58:19.662881 16513 tablet.cc:2369] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:19.747401 16798 raft_consensus.cc:482] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:19.747643 16798 raft_consensus.cc:504] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } }
I20260619 06:58:19.748900 16798 leader_election.cc:288] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f38358518a34a2b8910320239ebd9d5 (127.11.106.67:38731), f1fb4022ac8d4d4d9936e52953cfbfdc (127.11.106.66:33189)
I20260619 06:58:19.750995 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.749233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:47691 (local address 127.11.106.67:38731)
0619 06:58:19.749404 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.749410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.749619 (+ 209us) server_negotiation.cc:407] Connection header received
0619 06:58:19.749768 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.749771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.749820 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.749910 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.750605 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.750608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.750611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.750628 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.750646 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.750654 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.750746 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.750842 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.750878 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:58:19.751012 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.749138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:38731 (local address 127.11.106.65:47691)
0619 06:58:19.749503 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:19.749521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:19.749656 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:19.749930 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:19.749933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:19.749948 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:19.750451 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:19.750453 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:19.750462 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:19.750485 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:19.750741 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:19.750744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:19.750747 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:58:19.750831 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:58:19.750871 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":143,"threads_started":1}
I20260619 06:58:19.751173 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.749138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:33189 (local address 127.11.106.65:55657)
0619 06:58:19.749549 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:19.749572 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:19.749682 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:19.749961 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:19.749964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:19.749980 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:19.750595 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:19.750597 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:19.750607 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:19.750641 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:19.750909 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:19.750912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:19.750915 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 06:58:19.750989 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:58:19.751027 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":151,"thread_start_us":77,"threads_started":1}
I20260619 06:58:19.751156 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.749315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:55657 (local address 127.11.106.66:33189)
0619 06:58:19.749506 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.749510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.749653 (+ 143us) server_negotiation.cc:407] Connection header received
0619 06:58:19.749793 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.749799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.749835 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.749938 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.750770 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.750774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.750776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.750790 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.750817 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.750826 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.750925 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.750998 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.751036 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 06:58:19.751876 16728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c487b80d298048ddbeb36915ae5a367d" candidate_uuid: "67e7c1eca6e84ed6af1d65c206717cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f38358518a34a2b8910320239ebd9d5" is_pre_election: true
I20260619 06:58:19.751878 16597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c487b80d298048ddbeb36915ae5a367d" candidate_uuid: "67e7c1eca6e84ed6af1d65c206717cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" is_pre_election: true
I20260619 06:58:19.752182 16597 raft_consensus.cc:2454] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e7c1eca6e84ed6af1d65c206717cda in term 0.
I20260619 06:58:19.752182 16728 raft_consensus.cc:2454] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e7c1eca6e84ed6af1d65c206717cda in term 0.
I20260619 06:58:19.752693 16401 leader_election.cc:302] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67e7c1eca6e84ed6af1d65c206717cda, f1fb4022ac8d4d4d9936e52953cfbfdc; no voters:
I20260619 06:58:19.752938 16798 raft_consensus.cc:2781] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:19.753021 16798 raft_consensus.cc:482] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:19.753064 16798 raft_consensus.cc:3037] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:19.754063 16798 raft_consensus.cc:504] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } }
I20260619 06:58:19.754388 16798 leader_election.cc:288] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [CANDIDATE]: Term 1 election: Requested vote from peers 4f38358518a34a2b8910320239ebd9d5 (127.11.106.67:38731), f1fb4022ac8d4d4d9936e52953cfbfdc (127.11.106.66:33189)
I20260619 06:58:19.754705 16597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c487b80d298048ddbeb36915ae5a367d" candidate_uuid: "67e7c1eca6e84ed6af1d65c206717cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc"
I20260619 06:58:19.754725 16728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c487b80d298048ddbeb36915ae5a367d" candidate_uuid: "67e7c1eca6e84ed6af1d65c206717cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f38358518a34a2b8910320239ebd9d5"
I20260619 06:58:19.754837 16728 raft_consensus.cc:3037] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:19.754837 16597 raft_consensus.cc:3037] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:19.756064 16597 raft_consensus.cc:2454] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e7c1eca6e84ed6af1d65c206717cda in term 1.
I20260619 06:58:19.756064 16728 raft_consensus.cc:2454] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e7c1eca6e84ed6af1d65c206717cda in term 1.
I20260619 06:58:19.756354 16399 leader_election.cc:302] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f38358518a34a2b8910320239ebd9d5, 67e7c1eca6e84ed6af1d65c206717cda; no voters:
I20260619 06:58:19.756512 16798 raft_consensus.cc:2781] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:19.756757 16798 raft_consensus.cc:686] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 1 LEADER]: Becoming Leader. State: Replica: 67e7c1eca6e84ed6af1d65c206717cda, State: Running, Role: LEADER
I20260619 06:58:19.757086 16798 consensus_queue.cc:237] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } }
I20260619 06:58:19.760224 16323 catalog_manager.cc:5495] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda reported cstate change: term changed from 0 to 1, leader changed from <none> to 67e7c1eca6e84ed6af1d65c206717cda (127.11.106.65). New cstate: current_term: 1 leader_uuid: "67e7c1eca6e84ed6af1d65c206717cda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e7c1eca6e84ed6af1d65c206717cda" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 } health_report { overall_health: UNKNOWN } } }
W20260619 06:58:19.771404 16775 tablet.cc:2369] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:19.790027 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.788343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.11.106.65:36801)
0619 06:58:19.788503 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:19.788508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:19.788524 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:19.788674 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:19.788676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:19.788751 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:19.788854 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:19.789596 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:19.789600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:19.789603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:19.789614 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:19.789641 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:19.789649 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:19.789755 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:19.789837 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 06:58:19.789877 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:58:19.796614 16728 raft_consensus.cc:1261] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [term 1 FOLLOWER]: Refusing update from remote peer 67e7c1eca6e84ed6af1d65c206717cda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:19.796614 16597 raft_consensus.cc:1261] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [term 1 FOLLOWER]: Refusing update from remote peer 67e7c1eca6e84ed6af1d65c206717cda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:19.797585 16798 consensus_queue.cc:1035] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1fb4022ac8d4d4d9936e52953cfbfdc" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:19.797901 16802 consensus_queue.cc:1035] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f38358518a34a2b8910320239ebd9d5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:19.805706 16805 mvcc.cc:205] Tried to move back new op lower bound from 7298467019957698560 to 7298467019807838208. Current Snapshot: MvccSnapshot[applied={T|T < 7298467019957698560}]
I20260619 06:58:19.807571 16806 mvcc.cc:205] Tried to move back new op lower bound from 7298467019957698560 to 7298467019807838208. Current Snapshot: MvccSnapshot[applied={T|T < 7298467019957698560}]
I20260619 06:58:19.814795 16323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:19.814975 16323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:19.817487 16323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c6b2e35cbe34e54ab53a4da19d80dbe: Sending DeleteTablet for 3 replicas of tablet c487b80d298048ddbeb36915ae5a367d
I20260619 06:58:19.818300 16446 tablet_service.cc:1514] Processing DeleteTablet for tablet c487b80d298048ddbeb36915ae5a367d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:19 UTC) from {username='slave'} at 127.0.0.1:53988
I20260619 06:58:19.818300 16577 tablet_service.cc:1514] Processing DeleteTablet for tablet c487b80d298048ddbeb36915ae5a367d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:19 UTC) from {username='slave'} at 127.0.0.1:43378
I20260619 06:58:19.818323 16708 tablet_service.cc:1514] Processing DeleteTablet for tablet c487b80d298048ddbeb36915ae5a367d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:19 UTC) from {username='slave'} at 127.0.0.1:56978
I20260619 06:58:19.818846 16817 tablet_replica.cc:330] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: stopping tablet replica
I20260619 06:58:19.818997 16818 tablet_replica.cc:330] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda: stopping tablet replica
I20260619 06:58:19.819133 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 16384
I20260619 06:58:19.819306 16818 raft_consensus.cc:2229] T c487b80d298048ddbeb36915ae5a367d P 67e7c1eca6e84ed6af1d65c206717cda [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:19.820034 16816 tablet_replica.cc:330] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: stopping tablet replica
I20260619 06:58:19.820263 16817 raft_consensus.cc:2229] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:19.820317 16816 raft_consensus.cc:2229] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:19.820647 16816 raft_consensus.cc:2258] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:19.820664 16817 raft_consensus.cc:2258] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:19.822309 16816 ts_tablet_manager.cc:1905] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:19.822533 16817 ts_tablet_manager.cc:1905] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:19.824517 16816 ts_tablet_manager.cc:1918] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:19.824683 16816 log.cc:1196] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/c487b80d298048ddbeb36915ae5a367d
I20260619 06:58:19.825026 16816 ts_tablet_manager.cc:1939] T c487b80d298048ddbeb36915ae5a367d P f1fb4022ac8d4d4d9936e52953cfbfdc: Deleting consensus metadata
I20260619 06:58:19.825807 16817 ts_tablet_manager.cc:1918] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:19.825904 16817 log.cc:1196] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/c487b80d298048ddbeb36915ae5a367d
I20260619 06:58:19.826145 16310 catalog_manager.cc:4841] TS f1fb4022ac8d4d4d9936e52953cfbfdc (127.11.106.66:33189): tablet c487b80d298048ddbeb36915ae5a367d (table test-table [id=3fd8d6af2f8048ddaeedc8c989bff0c2]) successfully deleted
I20260619 06:58:19.826270 16817 ts_tablet_manager.cc:1939] T c487b80d298048ddbeb36915ae5a367d P 4f38358518a34a2b8910320239ebd9d5: Deleting consensus metadata
W20260619 06:58:19.827533 16308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:36801 (error 108)
I20260619 06:58:19.827680 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 16515
I20260619 06:58:19.828214 16308 catalog_manager.cc:4841] TS 4f38358518a34a2b8910320239ebd9d5 (127.11.106.67:38731): tablet c487b80d298048ddbeb36915ae5a367d (table test-table [id=3fd8d6af2f8048ddaeedc8c989bff0c2]) successfully deleted
I20260619 06:58:19.829062 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:19.828680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:36801 (local address 127.0.0.1:54004)
0619 06:58:19.828876 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:19.828957 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:36801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 06:58:19.829267 16308 catalog_manager.cc:4584] TS 67e7c1eca6e84ed6af1d65c206717cda (127.11.106.65:36801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c487b80d298048ddbeb36915ae5a367d: Network error: Client connection negotiation failed: client connection to 127.11.106.65:36801: connect: Connection refused (error 111)
I20260619 06:58:19.834045 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 16646
I20260619 06:58:19.841228 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 16293
2026-06-19T06:58:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1125 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 06:58:19.863049 11689 mini_oidc.cc:152] Starting JWKS server
I20260619 06:58:19.864975 11689 webserver.cc:460] Webserver started at https://127.0.0.1:40987/ using document root <none> and password file <none>
I20260619 06:58:19.865167 11689 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:58:19.865461 11689 webserver.cc:460] Webserver started at http://127.0.0.1:45387/ using document root <none> and password file <none>
2026-06-19T06:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:19Z Disabled control of system clock
I20260619 06:58:19.874627 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:37877
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:37877
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40987/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/testchainca.cert with env {}
W20260619 06:58:19.985677 16825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:19.985925 16825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:19.985965 16825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:19.990051 16825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:19.990106 16825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:19.990123 16825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:19.990168 16825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:19.990200 16825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:58:19.990222 16825 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:58:19.990258 16825 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40987/jwks.json
I20260619 06:58:19.995592 16825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:37877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:37877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40987/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:19.996701 16825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:19.997726 16825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:20.006338 16831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:20.006474 16825 server_base.cc:1034] running on GCE node
W20260619 06:58:20.006346 16833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:20.006356 16830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:20.007254 16825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:20.008399 16825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:20.009609 16825 hybrid_clock.cc:648] HybridClock initialized: now 1781852300009553 us; error 75 us; skew 500 ppm
I20260619 06:58:20.011618 16825 webserver.cc:460] Webserver started at http://127.11.106.126:39525/ using document root <none> and password file <none>
I20260619 06:58:20.012217 16825 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:20.012288 16825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:20.012508 16825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:20.014639 16825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "ea842ff2e59742929565848103cd85dc"
format_stamp: "Formatted at 2026-06-19 06:58:20 on dist-test-slave-10d0"
server_key: "273448b15083747c192e12073f2f2378"
server_key_iv: "f690776f74904abe2b72108246d8b59c"
server_key_version: "encryptionkey@0"
I20260619 06:58:20.015223 16825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "ea842ff2e59742929565848103cd85dc"
format_stamp: "Formatted at 2026-06-19 06:58:20 on dist-test-slave-10d0"
server_key: "273448b15083747c192e12073f2f2378"
server_key_iv: "f690776f74904abe2b72108246d8b59c"
server_key_version: "encryptionkey@0"
I20260619 06:58:20.018991 16825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 06:58:20.021526 16839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:20.022821 16825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:58:20.022948 16825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "ea842ff2e59742929565848103cd85dc"
format_stamp: "Formatted at 2026-06-19 06:58:20 on dist-test-slave-10d0"
server_key: "273448b15083747c192e12073f2f2378"
server_key_iv: "f690776f74904abe2b72108246d8b59c"
server_key_version: "encryptionkey@0"
I20260619 06:58:20.023056 16825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:20.050397 16825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:20.051210 16825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:20.051440 16825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:20.058712 16825 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:37877
I20260619 06:58:20.058723 16891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:37877 every 8 connection(s)
I20260619 06:58:20.059708 16825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:20.060688 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 16825
I20260619 06:58:20.060834 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:20.061117 11689 external_mini_cluster.cc:1428] Setting key 0d1e629b7aa95e563304382d15050952
I20260619 06:58:20.063517 16892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:20.069517 16892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc: Bootstrap starting.
I20260619 06:58:20.070477 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:20.062557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.11.106.126:37877)
0619 06:58:20.063219 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:20.063237 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:20.063276 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:58:20.063997 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:20.064017 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:20.064415 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:20.064822 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:20.065316 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.066337 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.067224 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.067697 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.067878 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:20.068801 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:20.068806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:20.068818 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:20.069201 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:20.069255 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:20.069276 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:20.069383 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:20.069499 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 06:58:20.069776 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":141,"threads_started":1}
I20260619 06:58:20.072202 16892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:20.073201 16892 log.cc:824] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:20.073909 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:58:20.075577 16892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc: No bootstrap required, opened a new log
I20260619 06:58:20.079191 16892 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea842ff2e59742929565848103cd85dc" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37877 } }
I20260619 06:58:20.079418 16892 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:20.079501 16892 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea842ff2e59742929565848103cd85dc, State: Initialized, Role: FOLLOWER
I20260619 06:58:20.079905 16892 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea842ff2e59742929565848103cd85dc" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37877 } }
I20260619 06:58:20.080030 16892 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:20.080107 16892 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:20.080226 16892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:20.081218 16892 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea842ff2e59742929565848103cd85dc" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37877 } }
I20260619 06:58:20.081584 16892 leader_election.cc:302] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea842ff2e59742929565848103cd85dc; no voters:
I20260619 06:58:20.081897 16892 leader_election.cc:288] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:20.082079 16897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:20.082347 16897 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [term 1 LEADER]: Becoming Leader. State: Replica: ea842ff2e59742929565848103cd85dc, State: Running, Role: LEADER
I20260619 06:58:20.082765 16897 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea842ff2e59742929565848103cd85dc" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37877 } }
I20260619 06:58:20.083190 16892 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:20.084399 16898 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea842ff2e59742929565848103cd85dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea842ff2e59742929565848103cd85dc" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37877 } } }
I20260619 06:58:20.084570 16898 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:20.084925 16899 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea842ff2e59742929565848103cd85dc. Latest consensus state: current_term: 1 leader_uuid: "ea842ff2e59742929565848103cd85dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea842ff2e59742929565848103cd85dc" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37877 } } }
I20260619 06:58:20.085006 16906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:20.085039 16899 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:20.088174 16906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:20.094296 16906 catalog_manager.cc:1347] Generated new cluster ID: 6a581929006e461ca3fbadb4c3b6f929
I20260619 06:58:20.094375 16906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:20.114253 16906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:20.115556 16906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:20.121973 16906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea842ff2e59742929565848103cd85dc: Generated new TSK 0
I20260619 06:58:20.122709 16906 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:58:20.157270 16922 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 06:58:20.157449 16922 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:58:20.158090 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:20.139634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40056 (local address 127.11.106.126:37877)
0619 06:58:20.139783 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:20.139788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:20.139977 (+ 189us) server_negotiation.cc:407] Connection header received
0619 06:58:20.140173 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:20.140176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:20.140236 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:20.140352 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:58:20.140620 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.141433 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.142271 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.142611 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.142715 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:20.143059 (+ 344us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:58:20.157652 (+ 14593us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:58:20.157945 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40056: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 06:58:20.158161 16895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40056: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 06:58:20.159440 16921 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:20.139487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:37877 (local address 127.0.0.1:40056)
0619 06:58:20.139896 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:20.139909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:20.140017 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:20.140376 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:20.140379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:20.140386 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:58:20.140483 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:20.140488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.141543 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.141546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:20.142155 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:20.142162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.142744 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.142746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:20.142817 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:20.142862 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:58:20.157913 (+ 15051us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:58:20.158126 (+ 213us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.106.126:37877: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":166,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.106.126:37877: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 06:58:20.160465 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 16825
2026-06-19T06:58:20Z chronyd exiting
I20260619 06:58:20.198035 11689 test_util.cc:175] -----------------------------------------------
I20260619 06:58:20.198148 11689 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781852273483449-11689-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 06:58:20.199306 11689 mini_oidc.cc:152] Starting JWKS server
I20260619 06:58:20.200177 11689 webserver.cc:460] Webserver started at https://127.0.0.1:33015/ using document root <none> and password file <none>
I20260619 06:58:20.200306 11689 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:58:20.200635 11689 webserver.cc:460] Webserver started at http://127.0.0.1:41093/ using document root <none> and password file <none>
2026-06-19T06:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:20Z Disabled control of system clock
I20260619 06:58:20.209618 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:40069
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:41413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:40069
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33015/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/testchainca.cert with env {}
W20260619 06:58:20.320421 16928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:20.320772 16928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:20.320818 16928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:20.324728 16928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:20.324782 16928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:20.324798 16928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:20.324815 16928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:20.324831 16928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:58:20.324846 16928 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:58:20.324858 16928 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33015/jwks.json
I20260619 06:58:20.330397 16928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:41413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:40069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33015/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:20.331445 16928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:20.332420 16928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:20.341533 16934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:20.341533 16936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:20.341591 16933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:20.341841 16928 server_base.cc:1034] running on GCE node
I20260619 06:58:20.342432 16928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:20.343580 16928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:20.344774 16928 hybrid_clock.cc:648] HybridClock initialized: now 1781852300344760 us; error 25 us; skew 500 ppm
I20260619 06:58:20.346567 16928 webserver.cc:460] Webserver started at http://127.11.106.126:37667/ using document root <none> and password file <none>
I20260619 06:58:20.347196 16928 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:20.347292 16928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:20.347522 16928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:20.349619 16928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "25046d090e4c4a1591ae7b674480e2cb"
format_stamp: "Formatted at 2026-06-19 06:58:20 on dist-test-slave-10d0"
server_key: "81f5e41ce898aad6d2f25567a511ea6d"
server_key_iv: "b0a1da044452d3155c1635f64714fc9a"
server_key_version: "encryptionkey@0"
I20260619 06:58:20.350226 16928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "25046d090e4c4a1591ae7b674480e2cb"
format_stamp: "Formatted at 2026-06-19 06:58:20 on dist-test-slave-10d0"
server_key: "81f5e41ce898aad6d2f25567a511ea6d"
server_key_iv: "b0a1da044452d3155c1635f64714fc9a"
server_key_version: "encryptionkey@0"
I20260619 06:58:20.353914 16928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:58:20.356776 16942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:20.358081 16928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:20.358237 16928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "25046d090e4c4a1591ae7b674480e2cb"
format_stamp: "Formatted at 2026-06-19 06:58:20 on dist-test-slave-10d0"
server_key: "81f5e41ce898aad6d2f25567a511ea6d"
server_key_iv: "b0a1da044452d3155c1635f64714fc9a"
server_key_version: "encryptionkey@0"
I20260619 06:58:20.358377 16928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:20.390894 16928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:20.391708 16928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:20.391927 16928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:20.399219 16928 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:40069
I20260619 06:58:20.399225 16994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:40069 every 8 connection(s)
I20260619 06:58:20.400187 16928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:20.403553 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:20.406423 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 16928
I20260619 06:58:20.406574 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:20.406862 11689 external_mini_cluster.cc:1428] Setting key abdfce36c2b280fcf8d87f4d8f3bc047
I20260619 06:58:20.409826 16995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb: Bootstrap starting.
I20260619 06:58:20.412595 16995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:20.413403 16995 log.cc:824] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:20.415685 16995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb: No bootstrap required, opened a new log
I20260619 06:58:20.417555 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:20.408312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.11.106.126:40069)
0619 06:58:20.409030 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:20.409052 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:20.409095 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:58:20.409935 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:20.409959 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:20.410466 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:20.410842 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:20.411322 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.412812 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.413793 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.414399 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.414562 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:20.415410 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:20.415417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:20.415433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:20.415946 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:20.416018 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:20.416048 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:20.416157 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:20.416338 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 06:58:20.416648 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":106,"threads_started":1}
I20260619 06:58:20.420229 16995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25046d090e4c4a1591ae7b674480e2cb" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 40069 } }
I20260619 06:58:20.420485 16995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:20.420589 16995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25046d090e4c4a1591ae7b674480e2cb, State: Initialized, Role: FOLLOWER
I20260619 06:58:20.420719 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:58:20.421082 16995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25046d090e4c4a1591ae7b674480e2cb" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 40069 } }
I20260619 06:58:20.421253 16995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:20.421311 16995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:20.421396 16995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:20.422518 16995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25046d090e4c4a1591ae7b674480e2cb" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 40069 } }
I20260619 06:58:20.422875 16995 leader_election.cc:302] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25046d090e4c4a1591ae7b674480e2cb; no voters:
I20260619 06:58:20.423192 16995 leader_election.cc:288] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:20.423328 17000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:20.423568 17000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [term 1 LEADER]: Becoming Leader. State: Replica: 25046d090e4c4a1591ae7b674480e2cb, State: Running, Role: LEADER
I20260619 06:58:20.423895 17000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25046d090e4c4a1591ae7b674480e2cb" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 40069 } }
I20260619 06:58:20.424552 16995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:20.425741 17001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25046d090e4c4a1591ae7b674480e2cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25046d090e4c4a1591ae7b674480e2cb" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 40069 } } }
I20260619 06:58:20.425743 17002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25046d090e4c4a1591ae7b674480e2cb. Latest consensus state: current_term: 1 leader_uuid: "25046d090e4c4a1591ae7b674480e2cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25046d090e4c4a1591ae7b674480e2cb" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 40069 } } }
I20260619 06:58:20.425971 17002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:20.426003 17001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:20.426359 17009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:20.429459 17009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:20.435325 17009 catalog_manager.cc:1347] Generated new cluster ID: d207fa9beefb453391da3531ce49dbdd
I20260619 06:58:20.435405 17009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:20.458287 17009 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:20.459296 17009 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:20.466985 17009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25046d090e4c4a1591ae7b674480e2cb: Generated new TSK 0
I20260619 06:58:20.467888 17009 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:58:20.506083 17025 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 06:58:20.506197 17025 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:58:20.507004 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:20.483743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.11.106.126:40069)
0619 06:58:20.483890 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:20.483897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:20.484026 (+ 129us) server_negotiation.cc:407] Connection header received
0619 06:58:20.484206 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:20.484208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:20.484258 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:20.484346 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:58:20.484640 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.485212 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.486485 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.486738 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.486831 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:20.487205 (+ 374us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:58:20.506526 (+ 19321us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:58:20.506838 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 06:58:20.507078 16998 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 06:58:20.507107 17024 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:20.483574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:40069 (local address 127.0.0.1:37488)
0619 06:58:20.483929 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:20.483944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:20.484061 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:20.484369 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:20.484373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:20.484381 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:58:20.484496 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:20.484502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.485328 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.485332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:20.486334 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:20.486345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.486862 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.486866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:20.486963 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:20.486992 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:58:20.506761 (+ 19769us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:58:20.506933 (+ 172us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.106.126:40069: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":116,"threads_started":1}
I20260619 06:58:20.507534 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 16928
2026-06-19T06:58:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (347 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 06:58:20.547087 11689 mini_oidc.cc:152] Starting JWKS server
I20260619 06:58:20.547964 11689 webserver.cc:460] Webserver started at https://127.0.0.1:44365/ using document root <none> and password file <none>
I20260619 06:58:20.548094 11689 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:58:20.548390 11689 webserver.cc:460] Webserver started at http://127.0.0.1:37687/ using document root <none> and password file <none>
2026-06-19T06:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:20Z Disabled control of system clock
I20260619 06:58:20.557462 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46563
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:46563
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44365/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:58:20.671052 17031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:20.671340 17031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:20.671379 17031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:20.675494 17031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:20.675563 17031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:20.675580 17031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:20.675596 17031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:20.675611 17031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:58:20.675668 17031 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:58:20.675695 17031 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44365/jwks.json
I20260619 06:58:20.680636 17031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:46563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44365/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:20.681834 17031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:20.683352 17031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:20.692456 17036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:20.692349 17037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:20.692342 17039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:20.692342 17031 server_base.cc:1034] running on GCE node
I20260619 06:58:20.693142 17031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:20.694221 17031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:20.695389 17031 hybrid_clock.cc:648] HybridClock initialized: now 1781852300695358 us; error 44 us; skew 500 ppm
I20260619 06:58:20.697292 17031 webserver.cc:460] Webserver started at http://127.11.106.126:36023/ using document root <none> and password file <none>
I20260619 06:58:20.697892 17031 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:20.697954 17031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:20.698176 17031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:20.700326 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "15b60c4c017e40b7bcac714d33588287"
format_stamp: "Formatted at 2026-06-19 06:58:20 on dist-test-slave-10d0"
server_key: "a85638977bde6bd49c57fb115f9521de"
server_key_iv: "9a974a3ff4962ff8704fbf5c06ab3a96"
server_key_version: "encryptionkey@0"
I20260619 06:58:20.700953 17031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "15b60c4c017e40b7bcac714d33588287"
format_stamp: "Formatted at 2026-06-19 06:58:20 on dist-test-slave-10d0"
server_key: "a85638977bde6bd49c57fb115f9521de"
server_key_iv: "9a974a3ff4962ff8704fbf5c06ab3a96"
server_key_version: "encryptionkey@0"
I20260619 06:58:20.704752 17031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:58:20.707193 17045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:20.708498 17031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:20.708665 17031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "15b60c4c017e40b7bcac714d33588287"
format_stamp: "Formatted at 2026-06-19 06:58:20 on dist-test-slave-10d0"
server_key: "a85638977bde6bd49c57fb115f9521de"
server_key_iv: "9a974a3ff4962ff8704fbf5c06ab3a96"
server_key_version: "encryptionkey@0"
I20260619 06:58:20.708793 17031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:20.724957 17031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:20.725715 17031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:20.725935 17031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:20.733541 17031 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:46563
I20260619 06:58:20.733548 17097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:46563 every 8 connection(s)
I20260619 06:58:20.734578 17031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:20.738018 17098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:20.743348 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 17031
I20260619 06:58:20.743516 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:20.743794 11689 external_mini_cluster.cc:1428] Setting key 827c12bd51f441feb67dd13b75bf0bf4
I20260619 06:58:20.744208 17098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287: Bootstrap starting.
I20260619 06:58:20.747642 17098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:20.748513 17098 log.cc:824] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:20.750633 17098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287: No bootstrap required, opened a new log
I20260619 06:58:20.753535 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:20.745138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35492 (local address 127.11.106.126:46563)
0619 06:58:20.745800 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:20.745822 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:20.745862 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:58:20.746546 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:20.746566 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:20.747013 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:20.747385 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:20.747945 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.749122 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.750073 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.750565 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.750747 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:20.751654 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:20.751659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:20.751674 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:20.752122 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:20.752178 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:20.752198 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:20.752315 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:20.752478 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 06:58:20.752779 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":119,"threads_started":1}
I20260619 06:58:20.754667 17098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b60c4c017e40b7bcac714d33588287" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46563 } }
I20260619 06:58:20.754932 17098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:20.755018 17098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b60c4c017e40b7bcac714d33588287, State: Initialized, Role: FOLLOWER
I20260619 06:58:20.755555 17098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b60c4c017e40b7bcac714d33588287" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46563 } }
I20260619 06:58:20.755726 17098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:20.755807 17098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:20.755919 17098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:20.756485 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:58:20.757128 17098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b60c4c017e40b7bcac714d33588287" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46563 } }
I20260619 06:58:20.757548 17098 leader_election.cc:302] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15b60c4c017e40b7bcac714d33588287; no voters:
I20260619 06:58:20.757920 17098 leader_election.cc:288] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:20.758031 17103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:20.758272 17103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [term 1 LEADER]: Becoming Leader. State: Replica: 15b60c4c017e40b7bcac714d33588287, State: Running, Role: LEADER
I20260619 06:58:20.758589 17103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b60c4c017e40b7bcac714d33588287" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46563 } }
I20260619 06:58:20.759255 17098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:20.760201 17105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15b60c4c017e40b7bcac714d33588287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b60c4c017e40b7bcac714d33588287" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46563 } } }
I20260619 06:58:20.760358 17105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:20.760793 17104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15b60c4c017e40b7bcac714d33588287. Latest consensus state: current_term: 1 leader_uuid: "15b60c4c017e40b7bcac714d33588287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b60c4c017e40b7bcac714d33588287" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46563 } } }
I20260619 06:58:20.760912 17104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:20.761224 17112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:20.764127 17112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:20.770443 17112 catalog_manager.cc:1347] Generated new cluster ID: 75c4e73652704f5c9b2f7fc261541dbc
I20260619 06:58:20.770563 17112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:20.790879 17112 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:20.792025 17112 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:20.799683 17112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15b60c4c017e40b7bcac714d33588287: Generated new TSK 0
I20260619 06:58:20.800601 17112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:58:20.815418 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:20.810010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.11.106.126:46563)
0619 06:58:20.810165 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:20.810171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:20.810324 (+ 153us) server_negotiation.cc:407] Connection header received
0619 06:58:20.810487 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:20.810489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:20.810535 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:20.810626 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:58:20.810879 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.811411 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.812770 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.813075 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.813168 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:20.813540 (+ 372us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:58:20.815044 (+ 1504us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:58:20.815278 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 06:58:20.815486 17101 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 06:58:20.817332 17127 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:20.809918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46563 (local address 127.0.0.1:35504)
0619 06:58:20.810220 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:20.810235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:20.810366 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:20.810647 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:20.810650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:20.810658 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:58:20.810775 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:20.810782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.811514 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.811518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:20.812653 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:20.812664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:20.813207 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:20.813211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:20.813307 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:20.813336 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:58:20.815224 (+ 1888us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:58:20.817144 (+ 1920us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.106.126:46563: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1711,"thread_start_us":108,"threads_started":1}
I20260619 06:58:20.817929 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 17031
E20260619 06:58:20.826344 17128 webserver.cc:533] Webserver: sslize failed:
E20260619 06:58:20.826414 17128 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T06:58:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (310 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T06:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:20Z Disabled control of system clock
I20260619 06:58:20.866158 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38897
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:38897
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/testchainca.cert with env {}
W20260619 06:58:20.979056 17132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:20.979363 17132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:20.979405 17132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:20.983494 17132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:20.983563 17132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:20.983579 17132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:20.983597 17132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:20.983613 17132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:58:20.983630 17132 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:58:20.983644 17132 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 06:58:20.989221 17132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:38897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:20.990283 17132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:20.991295 17132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:21.000631 17138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:21.000677 17140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:21.000640 17137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:21.000828 17132 server_base.cc:1034] running on GCE node
I20260619 06:58:21.001556 17132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:21.002705 17132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:21.003892 17132 hybrid_clock.cc:648] HybridClock initialized: now 1781852301003853 us; error 49 us; skew 500 ppm
I20260619 06:58:21.005973 17132 webserver.cc:460] Webserver started at http://127.11.106.126:43677/ using document root <none> and password file <none>
I20260619 06:58:21.006752 17132 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:21.006865 17132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:21.007181 17132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:21.010417 17132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "31e31d039b3948558b425082cb457be0"
format_stamp: "Formatted at 2026-06-19 06:58:21 on dist-test-slave-10d0"
server_key: "f9d7a320699c80504f6b2cf7ba783c50"
server_key_iv: "22edf5d993ffaad4e9cbe10e2f5bcf90"
server_key_version: "encryptionkey@0"
I20260619 06:58:21.011268 17132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "31e31d039b3948558b425082cb457be0"
format_stamp: "Formatted at 2026-06-19 06:58:21 on dist-test-slave-10d0"
server_key: "f9d7a320699c80504f6b2cf7ba783c50"
server_key_iv: "22edf5d993ffaad4e9cbe10e2f5bcf90"
server_key_version: "encryptionkey@0"
I20260619 06:58:21.016635 17132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 06:58:21.019361 17146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:21.020910 17132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:58:21.021061 17132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "31e31d039b3948558b425082cb457be0"
format_stamp: "Formatted at 2026-06-19 06:58:21 on dist-test-slave-10d0"
server_key: "f9d7a320699c80504f6b2cf7ba783c50"
server_key_iv: "22edf5d993ffaad4e9cbe10e2f5bcf90"
server_key_version: "encryptionkey@0"
I20260619 06:58:21.021195 17132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:21.036091 17132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:21.036991 17132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:21.037209 17132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:21.044895 17132 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:38897
I20260619 06:58:21.044937 17198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:38897 every 8 connection(s)
I20260619 06:58:21.045977 17132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:21.049232 17199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:21.052090 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 17132
I20260619 06:58:21.052232 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:21.052531 11689 external_mini_cluster.cc:1428] Setting key d3fd890a43b6aa7a654106dd9052167a
I20260619 06:58:21.055969 17199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0: Bootstrap starting.
I20260619 06:58:21.058570 17199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:21.059325 17199 log.cc:824] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:21.061674 17199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0: No bootstrap required, opened a new log
I20260619 06:58:21.062544 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:21.054081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.11.106.126:38897)
0619 06:58:21.054835 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:21.054853 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:21.054897 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:58:21.055586 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:21.055605 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:21.056068 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:21.056487 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:21.057002 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:21.057998 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:21.059012 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:21.059550 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:21.059729 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:21.060640 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:21.060646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:21.060661 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:21.061057 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:21.061113 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:21.061134 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:21.061231 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:21.061430 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 06:58:21.061766 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":136,"threads_started":1}
I20260619 06:58:21.065745 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:58:21.066030 17199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e31d039b3948558b425082cb457be0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38897 } }
I20260619 06:58:21.066257 17199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:21.066344 17199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e31d039b3948558b425082cb457be0, State: Initialized, Role: FOLLOWER
I20260619 06:58:21.066874 17199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e31d039b3948558b425082cb457be0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38897 } }
I20260619 06:58:21.067032 17199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:21.067097 17199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:21.067194 17199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:21.068226 17199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e31d039b3948558b425082cb457be0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38897 } }
I20260619 06:58:21.068524 17199 leader_election.cc:302] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31e31d039b3948558b425082cb457be0; no voters:
I20260619 06:58:21.068889 17199 leader_election.cc:288] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:21.068971 17204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:21.069248 17204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [term 1 LEADER]: Becoming Leader. State: Replica: 31e31d039b3948558b425082cb457be0, State: Running, Role: LEADER
I20260619 06:58:21.069545 17204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e31d039b3948558b425082cb457be0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38897 } }
I20260619 06:58:21.070161 17199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:21.071228 17205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31e31d039b3948558b425082cb457be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e31d039b3948558b425082cb457be0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38897 } } }
I20260619 06:58:21.071369 17205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:21.071779 17213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:21.072791 17206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31e31d039b3948558b425082cb457be0. Latest consensus state: current_term: 1 leader_uuid: "31e31d039b3948558b425082cb457be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e31d039b3948558b425082cb457be0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38897 } } }
I20260619 06:58:21.072893 17206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:21.075163 17213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:21.081029 17213 catalog_manager.cc:1347] Generated new cluster ID: d4a9d73012ba49d7b8ed1e6de9eaa21b
I20260619 06:58:21.081116 17213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:21.090238 17213 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:21.091293 17213 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:21.097011 17213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31e31d039b3948558b425082cb457be0: Generated new TSK 0
I20260619 06:58:21.097767 17213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:58:21.597820 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:21.109533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.11.106.126:38897)
0619 06:58:21.109684 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:21.109689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:21.109846 (+ 157us) server_negotiation.cc:407] Connection header received
0619 06:58:21.110023 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:21.110027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:21.110085 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:21.110210 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:58:21.110481 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:21.111306 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:21.112273 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:21.112658 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:21.112779 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:21.113090 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:58:21.597405 (+484315us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:58:21.597699 (+ 294us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49876: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 06:58:21.597926 17202 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49876: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 06:58:21.598102 17228 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:21.109400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38897 (local address 127.0.0.1:49876)
0619 06:58:21.109759 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:21.109772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:21.109886 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:21.110251 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:21.110254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:21.110263 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:58:21.110368 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:21.110373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:21.111440 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:21.111442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:21.112154 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:21.112161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:21.112795 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:21.112798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:21.112865 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:21.112892 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:58:21.597661 (+484769us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:58:21.597873 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.106.126:38897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":119,"threads_started":1}
I20260619 06:58:21.598752 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 17132
2026-06-19T06:58:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (760 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:21 dist-test-slave-10d0 krb5kdc[17240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:21 dist-test-slave-10d0 krb5kdc[17240](info): set up 2 sockets
Jun 19 06:58:21 dist-test-slave-10d0 krb5kdc[17240](info): commencing operation
krb5kdc: starting...
W20260619 06:58:22.901391 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:22 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:23.028340 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45991
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:45991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:23.140196 17253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:23.140523 17253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:23.140625 17253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:23.144641 17253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:23.144742 17253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:23.144780 17253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:23.144842 17253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:23.144898 17253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:23.150286 17253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:45991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:23.151473 17253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:23.152603 17253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:23.161192 17261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:23.161192 17258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:23.161211 17259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:23.161620 17253 server_base.cc:1034] running on GCE node
I20260619 06:58:23.162153 17253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:23.163273 17253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:23.164463 17253 hybrid_clock.cc:648] HybridClock initialized: now 1781852303164446 us; error 28 us; skew 500 ppm
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:23.171027 17253 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:23.172181 17253 webserver.cc:460] Webserver started at http://127.11.106.126:43093/ using document root <none> and password file <none>
I20260619 06:58:23.172932 17253 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:23.173003 17253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:23.173250 17253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:23.175519 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "5796b9cdea52406da548a9ef894705ca"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "1b4cdedf5e7ab23be8823f23f297227b"
server_key_iv: "de8ba62279e9a959ac235545d0cf0a94"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.176155 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "5796b9cdea52406da548a9ef894705ca"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "1b4cdedf5e7ab23be8823f23f297227b"
server_key_iv: "de8ba62279e9a959ac235545d0cf0a94"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.180513 17253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:58:23.183351 17268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.184912 17253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:23.185058 17253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "5796b9cdea52406da548a9ef894705ca"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "1b4cdedf5e7ab23be8823f23f297227b"
server_key_iv: "de8ba62279e9a959ac235545d0cf0a94"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.185177 17253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:23.201563 17253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:23.204969 17253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:23.205189 17253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:23.212625 17253 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:45991
I20260619 06:58:23.212652 17320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:45991 every 8 connection(s)
I20260619 06:58:23.213616 17253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:23.215030 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 17253
I20260619 06:58:23.215173 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:23.215476 11689 external_mini_cluster.cc:1428] Setting key 3166f4f574509811c2a81509d8bd0851
I20260619 06:58:23.216950 17321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:23.222739 17321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca: Bootstrap starting.
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:23.225129 17321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:23.225901 17321 log.cc:824] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:23.227777 17321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca: No bootstrap required, opened a new log
I20260619 06:58:23.229611 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:23.216972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.11.106.126:45991)
0619 06:58:23.217436 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:23.217453 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:23.217491 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:58:23.218119 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:23.218135 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:23.218489 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:23.218835 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:23.219651 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.220728 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.221630 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.222116 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.222319 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:23.224938 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:23.224943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:23.224956 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:23.224985 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:23.226784 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.227241 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.227252 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.227257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.227367 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.227661 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.227664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.227666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.228105 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:23.228323 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:23.228580 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 06:58:23.228888 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":152,"threads_started":1}
I20260619 06:58:23.231453 17321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5796b9cdea52406da548a9ef894705ca" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45991 } }
I20260619 06:58:23.231717 17321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:23.231776 17321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5796b9cdea52406da548a9ef894705ca, State: Initialized, Role: FOLLOWER
I20260619 06:58:23.232182 17321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5796b9cdea52406da548a9ef894705ca" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45991 } }
I20260619 06:58:23.232301 17321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:23.232352 17321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:23.232419 17321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:23.233470 17321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5796b9cdea52406da548a9ef894705ca" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45991 } }
I20260619 06:58:23.233816 17321 leader_election.cc:302] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5796b9cdea52406da548a9ef894705ca; no voters:
I20260619 06:58:23.234181 17321 leader_election.cc:288] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:23.234313 17326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:23.234549 17326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [term 1 LEADER]: Becoming Leader. State: Replica: 5796b9cdea52406da548a9ef894705ca, State: Running, Role: LEADER
I20260619 06:58:23.234843 17326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5796b9cdea52406da548a9ef894705ca" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45991 } }
I20260619 06:58:23.235435 17321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:23.236478 17328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5796b9cdea52406da548a9ef894705ca. Latest consensus state: current_term: 1 leader_uuid: "5796b9cdea52406da548a9ef894705ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5796b9cdea52406da548a9ef894705ca" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45991 } } }
I20260619 06:58:23.236668 17328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:23.237039 17327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5796b9cdea52406da548a9ef894705ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5796b9cdea52406da548a9ef894705ca" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45991 } } }
I20260619 06:58:23.237138 17327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:23.237919 17335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:23.240464 17335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:23.245993 17335 catalog_manager.cc:1347] Generated new cluster ID: 84cef6742f4f4b41ad9cefd6a7135770
I20260619 06:58:23.246057 17335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:23.254848 17335 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:23.256063 17335 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:23.262840 17335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5796b9cdea52406da548a9ef894705ca: Generated new TSK 0
I20260619 06:58:23.263523 17335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:58:23.299588 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45991
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:23.416939 17347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:23.417205 17347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:23.417248 17347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:23.421187 17347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:23.421241 17347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:23.421299 17347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:23.426491 17347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:23.427544 17347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:23.428592 17347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:23.438828 17353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:23.438843 17352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:23.439018 17355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:23.439018 17347 server_base.cc:1034] running on GCE node
I20260619 06:58:23.439635 17347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:23.440362 17347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:23.441558 17347 hybrid_clock.cc:648] HybridClock initialized: now 1781852303441531 us; error 49 us; skew 500 ppm
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:23.444499 17347 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:58:23.445483 17347 webserver.cc:460] Webserver started at http://127.11.106.65:42029/ using document root <none> and password file <none>
I20260619 06:58:23.446084 17347 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:23.446182 17347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:23.446426 17347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:23.448515 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "bd881ecec12448a2b7ed2869095999e1"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "0d14795b22919237aec9abaad1832492"
server_key_iv: "60492f19382998f3af0aeee6d1fc43e1"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.449172 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "bd881ecec12448a2b7ed2869095999e1"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "0d14795b22919237aec9abaad1832492"
server_key_iv: "60492f19382998f3af0aeee6d1fc43e1"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.453003 17347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:58:23.455443 17362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.456861 17347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:23.456975 17347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "bd881ecec12448a2b7ed2869095999e1"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "0d14795b22919237aec9abaad1832492"
server_key_iv: "60492f19382998f3af0aeee6d1fc43e1"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.457083 17347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:23.470243 17347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:23.473435 17347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:23.473591 17347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:23.474123 17347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:23.474901 17347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:23.474946 17347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.474984 17347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:23.475004 17347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.484337 17347 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:39153
I20260619 06:58:23.484360 17475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:39153 every 8 connection(s)
I20260619 06:58:23.485313 17347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:23.486727 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 17347
I20260619 06:58:23.486878 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:23.487139 11689 external_mini_cluster.cc:1428] Setting key 273e537108bbb81d84e38180fba90eb8
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 06:58:23.501503 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:23.487368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:43007 (local address 127.11.106.126:45991)
0619 06:58:23.487528 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:23.487533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:23.488492 (+ 959us) server_negotiation.cc:407] Connection header received
0619 06:58:23.490043 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:23.490046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:23.490113 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:23.490198 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:23.491813 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.492766 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.493977 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.494447 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.494594 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:23.497597 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:23.497604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:23.497610 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:23.497645 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:23.499038 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.499629 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.499633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.499635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.499720 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.500133 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.500137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.500146 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.500396 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:23.500478 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:23.501102 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 06:58:23.501253 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:58:23.502243 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:23.487834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45991 (local address 127.11.106.65:43007)
0619 06:58:23.488320 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:23.488386 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:23.489825 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:23.490424 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:23.490435 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:23.490960 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:23.491645 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:23.491660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.492925 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.492932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:23.493830 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:23.493840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.494618 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.494621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:23.494811 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:23.495468 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:23.495485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:23.497333 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:23.499148 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:23.499153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:23.499166 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:23.499521 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:23.499808 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:23.499810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:23.499811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:23.499990 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:23.500513 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:23.500520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:23.500841 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 06:58:23.501012 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 06:58:23.501274 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":139,"threads_started":1}
I20260619 06:58:23.503582 17476 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45991
I20260619 06:58:23.503866 17476 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:23.504374 17476 heartbeater.cc:511] Master 127.11.106.126:45991 requested a full tablet report, sending...
I20260619 06:58:23.506165 17285 ts_manager.cc:194] Registered new tserver with Master: bd881ecec12448a2b7ed2869095999e1 (127.11.106.65:39153)
I20260619 06:58:23.507836 17285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:43007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:58:23.517812 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45991
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:23.630281 17481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:23.630537 17481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:23.630578 17481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:23.634526 17481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:23.634574 17481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:23.634739 17481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:23.640275 17481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:23.641389 17481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:23.642442 17481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:23.652000 17489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:23.652010 17486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:23.652093 17481 server_base.cc:1034] running on GCE node
W20260619 06:58:23.652007 17487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:23.652707 17481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:23.653383 17481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:23.654572 17481 hybrid_clock.cc:648] HybridClock initialized: now 1781852303654550 us; error 31 us; skew 500 ppm
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:23.657322 17481 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:58:23.658223 17481 webserver.cc:460] Webserver started at http://127.11.106.66:36753/ using document root <none> and password file <none>
I20260619 06:58:23.658814 17481 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:23.658916 17481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:23.659147 17481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:23.661383 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "56502e54c49e46e19192c8153cf1ebac"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "579ba20839390c94b1c240f1dbb6329d"
server_key_iv: "2af2053087e4d1b48a5bf51b82dc0a2c"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.661970 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "56502e54c49e46e19192c8153cf1ebac"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "579ba20839390c94b1c240f1dbb6329d"
server_key_iv: "2af2053087e4d1b48a5bf51b82dc0a2c"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.665870 17481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:58:23.668429 17496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.669710 17481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:23.669848 17481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "56502e54c49e46e19192c8153cf1ebac"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "579ba20839390c94b1c240f1dbb6329d"
server_key_iv: "2af2053087e4d1b48a5bf51b82dc0a2c"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.669963 17481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:23.697265 17481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:23.700264 17481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:23.700479 17481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:23.701112 17481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:23.701982 17481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:23.702055 17481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.702152 17481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:23.702210 17481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.711385 17481 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:44343
I20260619 06:58:23.711406 17609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:44343 every 8 connection(s)
I20260619 06:58:23.712291 17481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:23.714491 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 17481
I20260619 06:58:23.714612 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:23.714917 11689 external_mini_cluster.cc:1428] Setting key 7db18822131326be9be86adbf19c18b7
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 06:58:23.727602 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:23.714167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:40437 (local address 127.11.106.126:45991)
0619 06:58:23.714352 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:23.714358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:23.715269 (+ 911us) server_negotiation.cc:407] Connection header received
0619 06:58:23.716853 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:23.716857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:23.716908 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:23.717005 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:23.718265 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.718796 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.720626 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.720936 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.721023 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:23.723945 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:23.723948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:23.723950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:23.723974 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:23.725524 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.726056 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.726059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.726060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.726138 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.726459 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.726461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.726463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.726634 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:23.726738 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:23.727305 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 06:58:23.727411 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:58:23.728798 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:23.714575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45991 (local address 127.11.106.66:40437)
0619 06:58:23.715093 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:23.715147 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:23.716632 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:23.717160 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:23.717166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:23.717516 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:23.718077 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:23.718101 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.718922 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.718929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:23.720458 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:23.720473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.721106 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.721111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:23.721339 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:23.721997 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:23.722025 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:23.723765 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:23.725637 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:23.725643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:23.725655 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:23.725935 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:23.726225 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:23.726228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:23.726229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:23.726377 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:23.726732 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:23.726737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:23.727082 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 06:58:23.727250 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 06:58:23.727636 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":136,"threads_started":1}
I20260619 06:58:23.730211 17610 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45991
I20260619 06:58:23.730512 17610 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:23.731108 17610 heartbeater.cc:511] Master 127.11.106.126:45991 requested a full tablet report, sending...
I20260619 06:58:23.732396 17285 ts_manager.cc:194] Registered new tserver with Master: 56502e54c49e46e19192c8153cf1ebac (127.11.106.66:44343)
I20260619 06:58:23.733371 17285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:40437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:58:23.745342 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45991
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:23.858476 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:23.858777 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:23.858824 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:23.862936 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:23.863004 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:23.863068 17615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:23.868378 17615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:23.869496 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:23.870500 17615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:23.880623 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:23.880720 17615 server_base.cc:1034] running on GCE node
W20260619 06:58:23.880685 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:23.880623 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:23.881415 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:23.882051 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:23.883210 17615 hybrid_clock.cc:648] HybridClock initialized: now 1781852303883196 us; error 30 us; skew 500 ppm
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:23.885968 17615 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:58:23.886790 17615 webserver.cc:460] Webserver started at http://127.11.106.67:40177/ using document root <none> and password file <none>
I20260619 06:58:23.887351 17615 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:23.887440 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:23.887667 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:23.889894 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "14a8eeaf5efa4620a543e844266ebbca"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "9380cd5c55d13c056f0484880bafcd72"
server_key_iv: "54071613b22120a4d12993fdb4bce843"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.890458 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "14a8eeaf5efa4620a543e844266ebbca"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "9380cd5c55d13c056f0484880bafcd72"
server_key_iv: "54071613b22120a4d12993fdb4bce843"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.894160 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:58:23.896688 17630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.897919 17615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:23.898056 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "14a8eeaf5efa4620a543e844266ebbca"
format_stamp: "Formatted at 2026-06-19 06:58:23 on dist-test-slave-10d0"
server_key: "9380cd5c55d13c056f0484880bafcd72"
server_key_iv: "54071613b22120a4d12993fdb4bce843"
server_key_version: "encryptionkey@0"
I20260619 06:58:23.898169 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:23.921026 17615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:23.924752 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:23.924968 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:23.925602 17615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:23.926450 17615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:23.926494 17615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.926564 17615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:23.926601 17615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:23.935252 17615 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:38549
I20260619 06:58:23.935323 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:38549 every 8 connection(s)
I20260619 06:58:23.936344 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:23.941891 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 17615
I20260619 06:58:23.942024 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:23.942320 11689 external_mini_cluster.cc:1428] Setting key b9aae7767ffb162f452eaea22185e758
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:23.950260 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:23.938280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:45241 (local address 127.11.106.126:45991)
0619 06:58:23.938448 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:23.938451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:23.939224 (+ 773us) server_negotiation.cc:407] Connection header received
0619 06:58:23.940624 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:23.940627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:23.940685 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:23.940779 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:23.942086 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.942680 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.944142 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.944446 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.944591 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:23.946955 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:23.946958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:23.946960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:23.946984 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:23.948403 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.948912 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.948915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.948916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.948992 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.949285 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.949287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.949288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.949455 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:23.949525 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:23.950026 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 06:58:23.950126 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:58:23.951068 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:23.938616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45991 (local address 127.11.106.67:45241)
0619 06:58:23.939053 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:23.939096 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:23.940376 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:23.940957 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:23.940963 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:23.941312 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:23.941929 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:23.941953 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.942819 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.942823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:23.944029 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:23.944040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.944682 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.944685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:23.944855 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:23.945370 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:23.945386 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:23.946801 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:23.948498 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:23.948502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:23.948514 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:23.948816 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:23.949075 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:23.949077 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:23.949078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:23.949200 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:23.949537 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:23.949542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:23.949789 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 06:58:23.949937 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 06:58:23.950180 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":107,"threads_started":1}
I20260619 06:58:23.952291 17744 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45991
I20260619 06:58:23.952572 17744 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:23.953073 17744 heartbeater.cc:511] Master 127.11.106.126:45991 requested a full tablet report, sending...
I20260619 06:58:23.954089 17285 ts_manager.cc:194] Registered new tserver with Master: 14a8eeaf5efa4620a543e844266ebbca (127.11.106.67:38549)
I20260619 06:58:23.954640 17285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:45241
I20260619 06:58:23.956578 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/test-admin.keytab.
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:23.973495 11689 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 06:58:23.976470 17751 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:23.975075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.11.106.65:39153)
0619 06:58:23.975376 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:23.975392 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:23.975416 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:58:23.975468 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:23.975473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:23.975662 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:23.975787 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:23.976269 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52572: BlockingRecv error: recv got EOF from 127.0.0.1:52572 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":94,"threads_started":1}
W20260619 06:58:23.976539 11689 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:23.977620 11689 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:23.978540 11689 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:58:23.987974 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:23.979348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.11.106.65:39153)
0619 06:58:23.979479 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:23.979484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:23.979588 (+ 104us) server_negotiation.cc:407] Connection header received
0619 06:58:23.979734 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:23.979737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:23.979787 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:23.979873 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:23.980419 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.981275 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.982103 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.982495 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.982623 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:23.984875 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:23.984884 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:23.984898 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:23.984919 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:23.986232 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.986667 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.986681 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.986683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.986758 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:23.987041 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:23.987045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:23.987047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:23.987281 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:23.987492 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:23.987688 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 06:58:23.987850 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 06:58:23.992049 17751 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:23.990891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.11.106.65:39153)
0619 06:58:23.991022 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:23.991026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:23.991150 (+ 124us) server_negotiation.cc:407] Connection header received
0619 06:58:23.991289 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:23.991293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:23.991331 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:23.991423 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:23.991927 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52602: BlockingRecv error: recv got EOF from 127.0.0.1:52602 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 06:58:23.992188 11689 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:23.993251 11689 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:23.994150 11689 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:58:23 dist-test-slave-10d0 krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:58:24.002768 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:23.994945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.11.106.65:39153)
0619 06:58:23.995063 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:23.995067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:23.995178 (+ 111us) server_negotiation.cc:407] Connection header received
0619 06:58:23.995338 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:23.995341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:23.995375 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:23.995455 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:23.995937 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.996436 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.997225 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:23.997449 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:23.997557 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:23.999791 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:23.999795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:23.999797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:23.999821 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:24.001181 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:24.001685 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:24.001688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:24.001690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:24.001765 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:24.002006 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:24.002009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:24.002010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:24.002169 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:24.002260 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:24.002457 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 06:58:24.002549 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 06:58:24.005573 17751 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:24.004601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.11.106.65:39153)
0619 06:58:24.004708 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:24.004712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:24.004805 (+ 93us) server_negotiation.cc:407] Connection header received
0619 06:58:24.004935 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:24.004938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:24.004981 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:24.005063 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:24.005460 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52614: BlockingRecv error: recv got EOF from 127.0.0.1:52614 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 06:58:24.005735 11689 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781852273483449-11689-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:58:24 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:24.006801 11689 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:58:24 dist-test-slave-10d0 krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:24.007691 11689 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:58:24 dist-test-slave-10d0 krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 06:58:24.016258 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:24.008503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.11.106.65:39153)
0619 06:58:24.008630 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:24.008634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:24.008793 (+ 159us) server_negotiation.cc:407] Connection header received
0619 06:58:24.008860 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:24.008862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:24.008903 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:24.008979 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:24.009476 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:24.010042 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:24.010812 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:24.011040 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:24.011145 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:24.013495 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:24.013498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:24.013500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:24.013522 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:24.014813 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:24.015209 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:24.015211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:24.015212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:24.015283 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:24.015517 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:24.015519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:24.015520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:24.015664 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:24.015772 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:24.015949 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 06:58:24.016070 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:58:24.017339 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 17347
I20260619 06:58:24.024247 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 17481
I20260619 06:58:24.029156 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 17615
I20260619 06:58:24.034451 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 17253
2026-06-19T06:58:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2188 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:24 dist-test-slave-10d0 krb5kdc[17764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:24 dist-test-slave-10d0 krb5kdc[17764](info): set up 2 sockets
Jun 19 06:58:24 dist-test-slave-10d0 krb5kdc[17764](info): commencing operation
krb5kdc: starting...
W20260619 06:58:25.077484 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:25 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.126.keytab.
Entry for principal oryx/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.126.keytab.
I20260619 06:58:25.203917 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:34885
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.126.keytab
--principal=oryx/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:25.314338 17777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:25.314631 17777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:25.314702 17777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:25.318691 17777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:25.318784 17777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:25.318823 17777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:25.318881 17777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:25.318938 17777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:25.324158 17777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.126.keytab
--principal=oryx/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:25.325276 17777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:25.326340 17777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:25.335500 17785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:25.335579 17777 server_base.cc:1034] running on GCE node
W20260619 06:58:25.335501 17782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:25.335505 17783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:25.336314 17777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:25.337504 17777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:25.338671 17777 hybrid_clock.cc:648] HybridClock initialized: now 1781852305338654 us; error 32 us; skew 500 ppm
Jun 19 06:58:25 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:25.341552 17777 init.cc:377] Logged in from keytab as oryx/127.11.106.126@KRBTEST.COM (short username oryx)
I20260619 06:58:25.342483 17777 webserver.cc:460] Webserver started at http://127.11.106.126:40907/ using document root <none> and password file <none>
I20260619 06:58:25.343065 17777 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:25.343125 17777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:25.343385 17777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:25.345497 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "7bffd54a4b224a75b78337fd20bc4c14"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "fb2dfccb32ad5bdc1742931354862beb"
server_key_iv: "fb423100fd95c20c7ef8fb72694fd93e"
server_key_version: "encryptionkey@0"
I20260619 06:58:25.346050 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "7bffd54a4b224a75b78337fd20bc4c14"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "fb2dfccb32ad5bdc1742931354862beb"
server_key_iv: "fb423100fd95c20c7ef8fb72694fd93e"
server_key_version: "encryptionkey@0"
I20260619 06:58:25.349964 17777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:58:25.352432 17792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:25.353781 17777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:25.353917 17777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "7bffd54a4b224a75b78337fd20bc4c14"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "fb2dfccb32ad5bdc1742931354862beb"
server_key_iv: "fb423100fd95c20c7ef8fb72694fd93e"
server_key_version: "encryptionkey@0"
I20260619 06:58:25.354023 17777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:25.366281 17777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:25.369076 17777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:25.369267 17777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:25.376225 17777 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:34885
I20260619 06:58:25.376505 17844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:34885 every 8 connection(s)
I20260619 06:58:25.377272 17777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:25.380017 17845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:25.380373 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 17777
I20260619 06:58:25.380517 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:25.380873 11689 external_mini_cluster.cc:1428] Setting key d107d6e1188771f63d68b9397eac01c1
I20260619 06:58:25.387221 17845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: Bootstrap starting.
I20260619 06:58:25.390055 17845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: Neither blocks nor log segments found. Creating new log.
Jun 19 06:58:25 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:25.390898 17845 log.cc:824] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:25.393000 17845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: No bootstrap required, opened a new log
I20260619 06:58:25.395539 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:25.382389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.11.106.126:34885)
0619 06:58:25.382813 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:25.382828 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:25.382869 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:58:25.383538 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:25.383556 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:25.383896 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:25.384173 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:25.385209 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.386208 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.387378 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.387891 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.388015 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:25.390836 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:25.390842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:25.390859 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:25.390890 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:25.392652 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:25.393157 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:25.393166 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:25.393174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:25.393296 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:25.393605 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:25.393607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:25.393609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:25.394030 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:25.394177 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:25.394492 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 06:58:25.394824 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":138,"threads_started":1}
I20260619 06:58:25.396956 17845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } }
I20260619 06:58:25.397212 17845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:25.397282 17845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bffd54a4b224a75b78337fd20bc4c14, State: Initialized, Role: FOLLOWER
I20260619 06:58:25.397725 17845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } }
I20260619 06:58:25.397920 17845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:25.398018 17845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:25.398116 17845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:25.398934 17845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } }
I20260619 06:58:25.399267 17845 leader_election.cc:302] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bffd54a4b224a75b78337fd20bc4c14; no voters:
I20260619 06:58:25.399518 17845 leader_election.cc:288] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:25.399677 17850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:25.399925 17850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 1 LEADER]: Becoming Leader. State: Replica: 7bffd54a4b224a75b78337fd20bc4c14, State: Running, Role: LEADER
I20260619 06:58:25.400250 17850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } }
I20260619 06:58:25.400838 17845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:25.402063 17852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bffd54a4b224a75b78337fd20bc4c14. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } }
I20260619 06:58:25.402077 17851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } }
I20260619 06:58:25.402211 17852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:25.402211 17851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
W20260619 06:58:25.405547 17865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:58:25.405658 17865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:58:25.405728 17860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:25.406473 17860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:25.412006 17860 catalog_manager.cc:1347] Generated new cluster ID: de7075af517749e9b4831371afb48188
I20260619 06:58:25.412081 17860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:25.423501 17860 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:25.424398 17860 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:25.431483 17860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: Generated new TSK 0
I20260619 06:58:25.432126 17860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab.
Entry for principal oryx/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab.
I20260619 06:58:25.467589 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:34885
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab
--principal=oryx/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:25.590258 17871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:25.590512 17871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:25.590553 17871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:25.594456 17871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:25.594506 17871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:25.594561 17871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:25.599525 17871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab
--principal=oryx/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:25.600449 17871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:25.601467 17871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:25.611279 17879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:25.611280 17876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:25.611307 17877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:25.611443 17871 server_base.cc:1034] running on GCE node
I20260619 06:58:25.611934 17871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:25.612617 17871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:25.613797 17871 hybrid_clock.cc:648] HybridClock initialized: now 1781852305613797 us; error 39 us; skew 500 ppm
Jun 19 06:58:25 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:25.616406 17871 init.cc:377] Logged in from keytab as oryx/127.11.106.65@KRBTEST.COM (short username oryx)
I20260619 06:58:25.617247 17871 webserver.cc:460] Webserver started at http://127.11.106.65:41981/ using document root <none> and password file <none>
I20260619 06:58:25.617820 17871 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:25.617884 17871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:25.618151 17871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:25.620186 17871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "0ea6eac2761c4cc5a736687209d01f3a"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "d1de3622916366519ba3157960648134"
server_key_iv: "b6e2a5815527d7efb10993aae42d7e0a"
server_key_version: "encryptionkey@0"
I20260619 06:58:25.620791 17871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "0ea6eac2761c4cc5a736687209d01f3a"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "d1de3622916366519ba3157960648134"
server_key_iv: "b6e2a5815527d7efb10993aae42d7e0a"
server_key_version: "encryptionkey@0"
I20260619 06:58:25.624446 17871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:25.626893 17886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:25.628079 17871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:25.628235 17871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "0ea6eac2761c4cc5a736687209d01f3a"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "d1de3622916366519ba3157960648134"
server_key_iv: "b6e2a5815527d7efb10993aae42d7e0a"
server_key_version: "encryptionkey@0"
I20260619 06:58:25.628351 17871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:25.639853 17871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:25.642977 17871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:25.643184 17871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:25.643800 17871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:25.644738 17871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:25.644807 17871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:25.644881 17871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:25.644923 17871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:25.654273 17871 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:36035
I20260619 06:58:25.654270 17999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:36035 every 8 connection(s)
I20260619 06:58:25.655157 17871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:25.664634 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 17871
I20260619 06:58:25.664783 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
Jun 19 06:58:25 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:25.665122 11689 external_mini_cluster.cc:1428] Setting key fbf41c08bb494c7bb1893f534a4eab1e
I20260619 06:58:25.669235 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:25.657124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:47585 (local address 127.11.106.126:34885)
0619 06:58:25.657249 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:25.657253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:25.658232 (+ 979us) server_negotiation.cc:407] Connection header received
0619 06:58:25.659630 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:25.659634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:25.659690 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:25.659801 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:25.660995 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.661542 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.662696 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.662936 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.663050 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:25.665677 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:25.665680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:25.665682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:25.665706 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:25.667153 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:25.667659 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:25.667664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:25.667666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:25.667749 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:25.668100 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:25.668103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:25.668105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:25.668284 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:25.668392 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:25.668944 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 06:58:25.669072 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 06:58:25.670264 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:25.657537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.65:47585)
0619 06:58:25.658005 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:25.658079 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:25.659417 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:25.659974 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:25.659980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:25.660351 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:25.660846 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:25.660859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.661677 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.661681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:25.662563 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:25.662573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.663085 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.663090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:25.663279 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:25.663856 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:25.663873 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:25.665501 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:25.667268 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:25.667274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:25.667289 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:25.667552 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:25.667848 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:25.667851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:25.667853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:25.667995 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:25.668390 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:25.668397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:25.668702 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 06:58:25.668901 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 06:58:25.669279 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":112,"threads_started":1}
I20260619 06:58:25.671684 18000 heartbeater.cc:347] Connected to a master server at 127.11.106.126:34885
I20260619 06:58:25.671948 18000 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:25.672431 18000 heartbeater.cc:511] Master 127.11.106.126:34885 requested a full tablet report, sending...
I20260619 06:58:25.673974 17809 ts_manager.cc:194] Registered new tserver with Master: 0ea6eac2761c4cc5a736687209d01f3a (127.11.106.65:36035)
WARNING: no policy specified for oryx/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 06:58:25.675132 17809 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:47585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab.
Entry for principal oryx/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab.
I20260619 06:58:25.693588 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:34885
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab
--principal=oryx/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:25.803520 18005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:25.803767 18005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:25.803808 18005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:25.808068 18005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:25.808127 18005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:25.808187 18005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:25.813382 18005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab
--principal=oryx/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:25.814366 18005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:25.815339 18005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:58:25.824769 18005 server_base.cc:1034] running on GCE node
W20260619 06:58:25.824683 18010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:25.824690 18011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:25.824680 18013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:25.825336 18005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:25.825953 18005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:25.827143 18005 hybrid_clock.cc:648] HybridClock initialized: now 1781852305827126 us; error 28 us; skew 500 ppm
Jun 19 06:58:25 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:25.829922 18005 init.cc:377] Logged in from keytab as oryx/127.11.106.66@KRBTEST.COM (short username oryx)
I20260619 06:58:25.830747 18005 webserver.cc:460] Webserver started at http://127.11.106.66:40897/ using document root <none> and password file <none>
I20260619 06:58:25.831385 18005 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:25.831470 18005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:25.831697 18005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:25.833861 18005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "5411bf416f874e9b9acfbaa08c340e20"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "b3efd287df7edcb87a62acdcdf2b047e"
server_key_iv: "52dc70a5fb348fa2179be9925d61a6a3"
server_key_version: "encryptionkey@0"
I20260619 06:58:25.834412 18005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "5411bf416f874e9b9acfbaa08c340e20"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "b3efd287df7edcb87a62acdcdf2b047e"
server_key_iv: "52dc70a5fb348fa2179be9925d61a6a3"
server_key_version: "encryptionkey@0"
I20260619 06:58:25.837955 18005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260619 06:58:25.840463 18020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:25.841776 18005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:58:25.841907 18005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "5411bf416f874e9b9acfbaa08c340e20"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "b3efd287df7edcb87a62acdcdf2b047e"
server_key_iv: "52dc70a5fb348fa2179be9925d61a6a3"
server_key_version: "encryptionkey@0"
I20260619 06:58:25.842015 18005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:25.856383 18005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:25.859896 18005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:25.860107 18005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:25.860761 18005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:25.861641 18005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:25.861712 18005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:25.861779 18005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:25.861824 18005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:25.870929 18005 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:42889
I20260619 06:58:25.870972 18133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:42889 every 8 connection(s)
I20260619 06:58:25.871874 18005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:25.879618 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 18005
I20260619 06:58:25.879796 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:25.880182 11689 external_mini_cluster.cc:1428] Setting key 99c5f8adf554f692504886f6f5012e54
Jun 19 06:58:25 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:25.886583 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:25.873875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:39651 (local address 127.11.106.126:34885)
0619 06:58:25.874041 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:25.874044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:25.874820 (+ 776us) server_negotiation.cc:407] Connection header received
0619 06:58:25.876223 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:25.876227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:25.876289 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:25.876410 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:25.877687 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.878242 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.879586 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.879910 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.880012 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:25.882884 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:25.882888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:25.882890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:25.882915 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:25.884418 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:25.884973 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:25.884978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:25.884979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:25.885064 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:25.885448 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:25.885452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:25.885453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:25.885653 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:25.885774 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:25.886297 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 06:58:25.886421 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:58:25.887442 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:25.874144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.66:39651)
0619 06:58:25.874654 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:25.874700 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:25.875988 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:25.876635 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:25.876643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:25.877015 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:25.877525 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:25.877540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.878408 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.878412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:25.879429 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:25.879443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:25.880097 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:25.880103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:25.880315 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:25.881065 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:25.881092 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:25.882707 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:25.884562 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:25.884568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:25.884586 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:25.884862 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:25.885190 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:25.885193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:25.885195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:25.885342 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:25.885769 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:25.885776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:25.886025 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 06:58:25.886235 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 06:58:25.886572 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":116,"threads_started":1}
I20260619 06:58:25.888655 18134 heartbeater.cc:347] Connected to a master server at 127.11.106.126:34885
I20260619 06:58:25.888916 18134 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:25.889393 18134 heartbeater.cc:511] Master 127.11.106.126:34885 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 06:58:25.890492 17809 ts_manager.cc:194] Registered new tserver with Master: 5411bf416f874e9b9acfbaa08c340e20 (127.11.106.66:42889)
I20260619 06:58:25.891078 17809 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:39651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab.
Entry for principal oryx/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab.
I20260619 06:58:25.907856 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:34885
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab
--principal=oryx/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:26.023838 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:26.024184 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:26.024286 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:26.028349 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:26.028404 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:26.028499 18139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:26.034107 18139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab
--principal=oryx/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:26.035146 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:26.036193 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:26.046046 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.046046 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.046067 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:26.046308 18139 server_base.cc:1034] running on GCE node
I20260619 06:58:26.046819 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:26.047485 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:26.048673 18139 hybrid_clock.cc:648] HybridClock initialized: now 1781852306048657 us; error 36 us; skew 500 ppm
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:26.051548 18139 init.cc:377] Logged in from keytab as oryx/127.11.106.67@KRBTEST.COM (short username oryx)
I20260619 06:58:26.052492 18139 webserver.cc:460] Webserver started at http://127.11.106.67:34933/ using document root <none> and password file <none>
I20260619 06:58:26.053086 18139 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:26.053194 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:26.053429 18139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:26.055425 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "7c0af17d04af45ca901f90a9ade63781"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "42560fb8d942a6f5da20ea468d72e423"
server_key_iv: "90a50f7b27a1e6b2e2ef0c99c6a06b21"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.055934 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "7c0af17d04af45ca901f90a9ade63781"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "42560fb8d942a6f5da20ea468d72e423"
server_key_iv: "90a50f7b27a1e6b2e2ef0c99c6a06b21"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.059952 18139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:58:26.062503 18154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.063887 18139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:58:26.064010 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "7c0af17d04af45ca901f90a9ade63781"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "42560fb8d942a6f5da20ea468d72e423"
server_key_iv: "90a50f7b27a1e6b2e2ef0c99c6a06b21"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.064126 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:26.082567 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:26.085894 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:26.086095 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:26.086694 18139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:26.087515 18139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:26.087584 18139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.087651 18139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:26.087698 18139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.096906 18139 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:39931
I20260619 06:58:26.096942 18267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:39931 every 8 connection(s)
I20260619 06:58:26.097810 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:26.104247 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 18139
I20260619 06:58:26.104380 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:26.104784 11689 external_mini_cluster.cc:1428] Setting key 687c2592f3688cdff00ac06ca758ce09
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:26.111940 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.099813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:40821 (local address 127.11.106.126:34885)
0619 06:58:26.099984 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.099989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.100793 (+ 804us) server_negotiation.cc:407] Connection header received
0619 06:58:26.102327 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.102330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.102376 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.102450 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.103580 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.104106 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.105748 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.106035 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.106127 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.108708 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.108711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.108713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.108737 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.110206 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.110648 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.110651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.110652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.110755 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.111037 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.111039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.111041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.111205 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.111276 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.111732 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.111824 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:58:26.112818 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.100080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.67:40821)
0619 06:58:26.100645 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.100688 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.102025 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.102610 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.102617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.102965 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.103441 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.103455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.104251 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.104255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.105576 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.105588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.106191 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.106195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.106351 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.106991 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.107013 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.108477 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.110304 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.110309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.110321 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.110556 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.110830 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.110832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.110834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.110956 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.111285 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.111290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.111505 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.111647 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.111894 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":116,"threads_started":1}
I20260619 06:58:26.113911 18268 heartbeater.cc:347] Connected to a master server at 127.11.106.126:34885
I20260619 06:58:26.114127 18268 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:26.114566 18268 heartbeater.cc:511] Master 127.11.106.126:34885 requested a full tablet report, sending...
I20260619 06:58:26.115530 17809 ts_manager.cc:194] Registered new tserver with Master: 7c0af17d04af45ca901f90a9ade63781 (127.11.106.67:39931)
I20260619 06:58:26.116066 17809 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:40821
I20260619 06:58:26.118643 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.106.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.125.keytab.
Entry for principal oryx/127.11.106.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.125.keytab.
I20260619 06:58:26.147606 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.125:39157
--webserver_interface=127.11.106.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.125.keytab
--principal=oryx/127.11.106.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:34885,127.11.106.125:39157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:26.263134 18273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:26.263399 18273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:26.263479 18273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:26.267653 18273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:26.267719 18273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:26.267740 18273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:26.267756 18273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:26.267810 18273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:26.273231 18273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:34885,127.11.106.125:39157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.125.keytab
--principal=oryx/127.11.106.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.125:39157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.106.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
W20260619 06:58:26.273453 18273 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.106.126:34885,127.11.106.125:39157'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 06:58:26.274365 18273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:26.275424 18273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:26.283931 18278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.283953 18279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:26.284046 18273 server_base.cc:1034] running on GCE node
W20260619 06:58:26.283931 18281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:26.284749 18273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:26.285858 18273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:26.287014 18273 hybrid_clock.cc:648] HybridClock initialized: now 1781852306286983 us; error 45 us; skew 500 ppm
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:26.289816 18273 init.cc:377] Logged in from keytab as oryx/127.11.106.125@KRBTEST.COM (short username oryx)
I20260619 06:58:26.290657 18273 webserver.cc:460] Webserver started at http://127.11.106.125:34665/ using document root <none> and password file <none>
I20260619 06:58:26.291177 18273 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:26.291239 18273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:26.291405 18273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:26.293540 18273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data/instance:
uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "91e50e158d066b7cddf22a4159a4b423"
server_key_iv: "b925f88aa8bf209aef489ac687cd30ac"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.294071 18273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal/instance:
uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "91e50e158d066b7cddf22a4159a4b423"
server_key_iv: "b925f88aa8bf209aef489ac687cd30ac"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.298080 18273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:58:26.300765 18288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.302316 18273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:26.302465 18273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal
uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "91e50e158d066b7cddf22a4159a4b423"
server_key_iv: "b925f88aa8bf209aef489ac687cd30ac"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.302589 18273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:26.318882 18273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:26.322494 18273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:26.322665 18273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:26.330011 18273 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.125:39157
I20260619 06:58:26.330330 18340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.125:39157 every 8 connection(s)
I20260619 06:58:26.331054 18273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
I20260619 06:58:26.334060 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 18273
I20260619 06:58:26.334112 18341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:26.338737 18341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.106.125@KRBTEST.COM
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.125@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:26.349041 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.335598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.11.106.125:39157)
0619 06:58:26.336038 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.336055 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.336090 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:58:26.336887 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.336906 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.337252 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.337589 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.338388 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.339449 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.340411 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.341334 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.341488 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.344224 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.344230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.344239 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.344266 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.346007 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.346469 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.346475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.346480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.346630 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.346921 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.346926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.346928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.347319 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.347511 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.347825 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.348092 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":138,"threads_started":1}
I20260619 06:58:26.350149 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.340463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.11.106.126:34885)
0619 06:58:26.340623 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.340629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.340879 (+ 250us) server_negotiation.cc:407] Connection header received
0619 06:58:26.341067 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.341072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.341124 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.341200 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.342087 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.342595 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.343847 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.344072 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.344186 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.346710 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.346714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.346719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.346749 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.348241 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.348761 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.348765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.348767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.348859 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.349139 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.349142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.349144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.349421 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.349518 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.349877 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.349988 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:58:26.350159 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.340323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:52576)
0619 06:58:26.340764 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.340811 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.340963 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.341271 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.341277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.341780 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.341950 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.341961 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.342733 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.342742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.343722 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.343738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.344221 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.344226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.344338 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.344970 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.344982 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.346590 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.348343 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.348353 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.348359 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.348647 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.348956 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.348958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.348960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.349034 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.349532 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.349537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.349633 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.349802 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.349983 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":100,"threads_started":1}
I20260619 06:58:26.351343 18341 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.125@KRBTEST.COM for oryx/127.11.106.125@KRBTEST.COM
I20260619 06:58:26.359058 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.351846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.0.0.1:45484)
0619 06:58:26.351926 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.351937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.352022 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.352237 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.352240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.352392 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.352464 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.352468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.353601 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.353604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.354096 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.354103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.354542 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.354545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.354657 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.355032 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.355043 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.356124 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.357567 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.357573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.357576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.357809 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.358077 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.358082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.358084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.358159 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.358515 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.358518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.358604 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.358708 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.358841 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":1}
I20260619 06:58:26.359058 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.351941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.11.106.125:39157)
0619 06:58:26.352030 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.352033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.352047 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:58:26.352102 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.352105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.352138 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.352238 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.352592 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.353481 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.354208 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.354439 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.354545 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.356260 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.356267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.356269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.356287 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.357464 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.357914 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.357917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.357920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.357990 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.358248 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.358253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.358254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.358419 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.358523 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.358750 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.358891 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":38}
I20260619 06:58:26.362037 18341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Bootstrap starting.
I20260619 06:58:26.364197 18341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:26.364946 18341 log.cc:824] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:26.366698 18341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: No bootstrap required, opened a new log
I20260619 06:58:26.369863 18341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } }
I20260619 06:58:26.370055 18341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:26.370090 18341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46bf7f0b36ab4d9a846c19ae6e4991e7, State: Initialized, Role: FOLLOWER
I20260619 06:58:26.370453 18341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } }
I20260619 06:58:26.370908 18347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } } }
I20260619 06:58:26.371047 18347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:58:26.371573 18341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:58:26.376588 18358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:58:26.376736 18358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:58:26.378556 18273 master_runner.cc:428] Detected that this master 46bf7f0b36ab4d9a846c19ae6e4991e7 is joining an existing cluster
I20260619 06:58:26.378625 18273 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.106.125:39157
I20260619 06:58:26.389140 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.381228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.11.106.125:39157)
0619 06:58:26.381326 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.381331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.381535 (+ 204us) server_negotiation.cc:407] Connection header received
0619 06:58:26.381666 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.381668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.381697 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.381749 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.382404 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.383152 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.384290 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.384666 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.384748 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.386100 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.386103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.386106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.386124 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.387613 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.388038 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.388041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.388043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.388102 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.388318 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.388321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.388322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.388508 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.388608 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.388885 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.388984 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260619 06:58:26.389808 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.380912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.0.0.1:45498)
0619 06:58:26.381460 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.381473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.381585 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.381860 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.381863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.382106 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.382293 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.382301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.383274 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.383277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.384172 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.384181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.384900 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.384903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.384991 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.385500 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.385515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.385983 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.387715 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.387728 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.387730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.387953 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.388181 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.388184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.388185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.388245 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.388663 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.388665 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.388751 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.389515 (+ 764us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.389630 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":34,"thread_start_us":92,"threads_started":1}
I20260619 06:58:26.389925 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.381187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.11.106.126:34885)
0619 06:58:26.381305 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.381310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.381377 (+ 67us) server_negotiation.cc:407] Connection header received
0619 06:58:26.381551 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.381554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.381618 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.381673 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.382474 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.383299 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.384505 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.384874 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.384952 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.386356 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.386362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.386365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.386393 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.388085 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.388593 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.388597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.388599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.388696 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.389005 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.389009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.389011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.389215 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.389286 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.389658 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.389771 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:58:26.390394 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.380812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:52588)
0619 06:58:26.381297 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.381311 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.381445 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.381826 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.381830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.382149 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.382337 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.382344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.383445 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.383448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.384371 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.384382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.385102 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.385105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.385209 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.385669 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.385689 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.386195 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.388208 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.388211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.388215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.388451 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.388812 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.388815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.388817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.388901 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.389367 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.389373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.389464 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.390050 (+ 586us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.390202 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":60,"thread_start_us":98,"threads_started":1}
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.126@KRBTEST.COM for oryx/127.11.106.125@KRBTEST.COM
I20260619 06:58:26.404350 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.395120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.11.106.125:39157)
0619 06:58:26.395242 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.395246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.395530 (+ 284us) server_negotiation.cc:407] Connection header received
0619 06:58:26.395737 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.395740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.395797 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.395894 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.396520 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.397047 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.397934 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.398149 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.398273 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.400607 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.400611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.400613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.400642 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.402279 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.402837 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.402841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.402843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.402952 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.403240 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.403243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.403246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.403523 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.403632 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.404066 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.404190 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:58:26.404516 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.395073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.0.0.1:45508)
0619 06:58:26.395416 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.395453 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.395610 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.395928 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.395934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.396300 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.396401 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.396407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.397161 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.397166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.397809 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.397818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.398279 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.398282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.398414 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.398936 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.398947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.400451 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.402387 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.402397 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.402404 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.402719 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.403035 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.403039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.403041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.403139 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.403647 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.403658 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.403817 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.403990 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.404235 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
I20260619 06:58:26.406286 17809 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.106.125:39157: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7"
member_type: NON_VOTER
last_known_addr {
host: "127.11.106.125"
port: 39157
}
attrs {
promote: true
}
}
dest_uuid: "7bffd54a4b224a75b78337fd20bc4c14"
cas_config_opid_index: -1
I20260619 06:58:26.407049 17809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } }
I20260619 06:58:26.409473 18369 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 46bf7f0b36ab4d9a846c19ae6e4991e7 (127.11.106.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } }
I20260619 06:58:26.409896 18315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:26.411314 18315 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Refusing update from remote peer 7bffd54a4b224a75b78337fd20bc4c14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 06:58:26.411394 18369 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.106.125:39157
I20260619 06:58:26.411727 18369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } }
I20260619 06:58:26.411938 18369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.11.106.124@KRBTEST.COM; defaulting to no policy
I20260619 06:58:26.412200 18273 master.cc:537] Master@127.11.106.125:39157 shutting down...
I20260619 06:58:26.412149 18369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:58:26.414288 17795 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 06:58:26.415189 17795 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 -> Peer 46bf7f0b36ab4d9a846c19ae6e4991e7 (127.11.106.125:39157): Couldn't send request to peer 46bf7f0b36ab4d9a846c19ae6e4991e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.124@KRBTEST.COM" created.
I20260619 06:58:26.417500 18273 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:26.417784 18273 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:26.417889 18273 tablet_replica.cc:330] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.124.keytab.
Entry for principal oryx/127.11.106.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.124.keytab.
I20260619 06:58:26.431777 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.124:43591
--webserver_interface=127.11.106.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.124.keytab
--principal=oryx/127.11.106.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:34885,127.11.106.125:39157,127.11.106.124:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:58:26.433938 18273 master.cc:552] Master@127.11.106.125:39157 shutdown complete.
I20260619 06:58:26.433995 18273 master_runner.cc:305] Clearing existing system tablet
I20260619 06:58:26.434247 18273 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:26.434303 18273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:26.435698 18273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:26.436656 18375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.436923 18273 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:26.437047 18273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:26.437079 18273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal
uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "91e50e158d066b7cddf22a4159a4b423"
server_key_iv: "b925f88aa8bf209aef489ac687cd30ac"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.439291 18273 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:26.441733 18273 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:58:26.441821 18273 log.cc:1196] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 06:58:26.442088 18273 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Deleting consensus metadata
I20260619 06:58:26.442256 18273 master_runner.cc:315] Copying system tablet from 127.11.106.126:34885
I20260619 06:58:26.443470 18273 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: tablet copy: Beginning tablet copy session from remote peer at address 127.11.106.126:34885
I20260619 06:58:26.452667 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.444171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.11.106.126:34885)
0619 06:58:26.444311 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.444316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.444587 (+ 271us) server_negotiation.cc:407] Connection header received
0619 06:58:26.444694 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.444698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.444753 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.444841 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.445357 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.446224 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.447395 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.447744 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.447847 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.449236 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.449240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.449243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.449270 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.450889 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.451403 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.451407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.451409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.451492 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.451774 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.451777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.451779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.451984 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.452080 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.452357 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.452473 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:58:26.452674 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.444050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:52594)
0619 06:58:26.444441 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.444455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.444600 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.444871 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.444874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.445106 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.445237 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.445244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.446346 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.446349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.447268 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.447279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.447862 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.447865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.447962 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.448531 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.448546 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.449069 (+ 523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.451016 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.451020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.451022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.451304 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.451585 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.451588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.451590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.451682 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.452078 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.452081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.452177 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.452289 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.452422 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":96,"threads_started":1}
I20260619 06:58:26.453817 17829 tablet_copy_service.cc:140] P 7bffd54a4b224a75b78337fd20bc4c14: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 46bf7f0b36ab4d9a846c19ae6e4991e7 ({username='oryx', principal='oryx/127.11.106.125@KRBTEST.COM'} at 127.0.0.1:52594)
I20260619 06:58:26.453998 17829 tablet_copy_service.cc:161] P 7bffd54a4b224a75b78337fd20bc4c14: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 46bf7f0b36ab4d9a846c19ae6e4991e7 at {username='oryx', principal='oryx/127.11.106.125@KRBTEST.COM'} at 127.0.0.1:52594: session id = 46bf7f0b36ab4d9a846c19ae6e4991e7-00000000000000000000000000000000
I20260619 06:58:26.455750 17829 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:58:26.457401 18273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:58:26.459697 18273 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: tablet copy: Starting download of 0 data blocks...
I20260619 06:58:26.459859 18273 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: tablet copy: Starting download of 1 WAL segments...
I20260619 06:58:26.461621 18273 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:58:26.463402 17829 tablet_copy_service.cc:342] P 7bffd54a4b224a75b78337fd20bc4c14: Request end of tablet copy session 46bf7f0b36ab4d9a846c19ae6e4991e7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.106.125@KRBTEST.COM'} at 127.0.0.1:52594
I20260619 06:58:26.463514 17829 tablet_copy_service.cc:433] P 7bffd54a4b224a75b78337fd20bc4c14: ending tablet copy session 46bf7f0b36ab4d9a846c19ae6e4991e7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 46bf7f0b36ab4d9a846c19ae6e4991e7
I20260619 06:58:26.470835 18273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:26.472626 18382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.472759 18383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.472815 18385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:26.473199 18273 server_base.cc:1034] running on GCE node
I20260619 06:58:26.473361 18273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:26.473637 18273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:26.481002 18273 hybrid_clock.cc:648] HybridClock initialized: now 1781852306480972 us; error 50 us; skew 500 ppm
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:26.482744 18273 init.cc:377] Logged in from keytab as oryx/127.11.106.125@KRBTEST.COM (short username oryx)
I20260619 06:58:26.483271 18273 webserver.cc:460] Webserver started at http://127.11.106.125:44221/ using document root <none> and password file <none>
I20260619 06:58:26.483470 18273 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:26.483513 18273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:26.485061 18273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:26.486212 18392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.486537 18273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:26.486634 18273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal
uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "91e50e158d066b7cddf22a4159a4b423"
server_key_iv: "b925f88aa8bf209aef489ac687cd30ac"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.486737 18273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:26.522347 18273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:26.524694 18273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:26.529717 18273 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.125:39157
I20260619 06:58:26.529747 18444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.125:39157 every 8 connection(s)
I20260619 06:58:26.530493 18273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
I20260619 06:58:26.533375 18445 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:58:26.534497 18445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Bootstrap starting.
I20260619 06:58:26.546372 18445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:58:26.546854 18445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Bootstrap complete.
I20260619 06:58:26.547413 18445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } }
I20260619 06:58:26.547540 18445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46bf7f0b36ab4d9a846c19ae6e4991e7, State: Initialized, Role: LEARNER
I20260619 06:58:26.547641 18445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } }
I20260619 06:58:26.547953 18448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } }
I20260619 06:58:26.548058 18445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:26.548136 18448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:58:26.552814 18459 catalog_manager.cc:1259] Loaded cluster ID: de7075af517749e9b4831371afb48188
I20260619 06:58:26.552944 18459 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: loading cluster ID for follower catalog manager: success
I20260619 06:58:26.555130 18459 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: acquiring CA information for follower catalog manager: success
I20260619 06:58:26.556727 18459 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:58:26.574867 18373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:26.575114 18373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:26.575153 18373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:26.579140 18373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:26.579196 18373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:26.579216 18373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:26.579232 18373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:26.579247 18373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:26.584471 18373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:34885,127.11.106.125:39157,127.11.106.124:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.124.keytab
--principal=oryx/127.11.106.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.124:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.106.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:26.585520 18373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:26.586483 18373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:26.594990 18465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.595013 18464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.594988 18467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:26.595336 18373 server_base.cc:1034] running on GCE node
I20260619 06:58:26.595839 18373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:26.596968 18373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:26.598124 18373 hybrid_clock.cc:648] HybridClock initialized: now 1781852306598109 us; error 25 us; skew 500 ppm
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:26.601109 18373 init.cc:377] Logged in from keytab as oryx/127.11.106.124@KRBTEST.COM (short username oryx)
I20260619 06:58:26.602038 18373 webserver.cc:460] Webserver started at http://127.11.106.124:44959/ using document root <none> and password file <none>
I20260619 06:58:26.602689 18373 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:26.602752 18373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:26.602990 18373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:26.605118 18373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data/instance:
uuid: "6f91ecc0c3184b5282e4146af78161b4"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "9664e9d48a11bd2de9e24e7ee4b05e3d"
server_key_iv: "478f390c7b7508478582a682ed3b8b52"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.605696 18373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal/instance:
uuid: "6f91ecc0c3184b5282e4146af78161b4"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "9664e9d48a11bd2de9e24e7ee4b05e3d"
server_key_iv: "478f390c7b7508478582a682ed3b8b52"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.609560 18373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:58:26.611980 18474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.613490 18373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:26.613626 18373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal
uuid: "6f91ecc0c3184b5282e4146af78161b4"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "9664e9d48a11bd2de9e24e7ee4b05e3d"
server_key_iv: "478f390c7b7508478582a682ed3b8b52"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.613729 18373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:26.632990 18373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:26.636049 18373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:26.636265 18373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:26.643718 18373 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.124:43591
I20260619 06:58:26.644019 18526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.124:43591 every 8 connection(s)
I20260619 06:58:26.644809 18373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
I20260619 06:58:26.647867 18527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:26.648881 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 18373
I20260619 06:58:26.652594 18527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.106.124@KRBTEST.COM
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.124@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:26.664491 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.654564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.11.106.126:34885)
0619 06:58:26.654679 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.654684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.655056 (+ 372us) server_negotiation.cc:407] Connection header received
0619 06:58:26.655280 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.655284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.655332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.655383 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.656388 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.657182 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.658477 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.658861 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.658944 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.661267 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.661271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.661275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.661303 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.662688 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.663169 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.663181 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.663183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.663262 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.663557 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.663560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.663562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.663791 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.663885 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.664215 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.664331 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:58:26.664680 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.650499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.11.106.124:43591)
0619 06:58:26.650947 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.650966 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.651040 (+ 74us) server_negotiation.cc:407] Connection header received
0619 06:58:26.651871 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.651895 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.652322 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.652724 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.653478 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.654936 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.655998 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.656779 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.656927 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.660029 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.660036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.660049 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.660079 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.661879 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.662310 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.662316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.662323 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.662426 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.662704 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.662709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.662711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.663080 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.663301 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.663544 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.663802 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
I20260619 06:58:26.664683 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.654412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:52596)
0619 06:58:26.654923 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.654974 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.655149 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.655553 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.655558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.655990 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.656223 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.656232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.657318 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.657321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.658314 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.658340 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.659011 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.659016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.659141 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.659750 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.659766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.661096 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.662818 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.662826 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.662832 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.663064 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.663354 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.663358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.663360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.663448 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.663885 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.663893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.664005 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.664143 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.664332 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":46,"spinlock_wait_cycles":4864,"thread_start_us":107,"threads_started":1}
I20260619 06:58:26.665962 18527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.124@KRBTEST.COM for oryx/127.11.106.125@KRBTEST.COM
I20260619 06:58:26.673714 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.666460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.0.0.1:45512)
0619 06:58:26.666757 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.666770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.666835 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.667149 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.667151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.667271 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.667356 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.667361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.668333 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.668336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.668921 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.668930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.669320 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.669322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.669392 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.669747 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.669757 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.670836 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.672468 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.672470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.672472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.672706 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.672953 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.672955 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.672956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.673062 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.673338 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.673340 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.673409 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.673484 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.673575 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260619 06:58:26.673720 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.666578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.11.106.125:39157)
0619 06:58:26.666823 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.666827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.666839 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:58:26.666924 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.666926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.667034 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.667118 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.667455 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.668241 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.669017 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.669227 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.669305 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.670953 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.670956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.670958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.670979 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.672380 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.672803 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.672805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.672806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.672875 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.673118 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.673120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.673121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.673266 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.673332 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.673518 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.673605 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260619 06:58:26.674476 18527 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:58:26.677861 18000 heartbeater.cc:503] Master 127.11.106.126:34885 was elected leader, sending a full tablet report...
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.124@KRBTEST.COM for oryx/127.11.106.124@KRBTEST.COM
I20260619 06:58:26.681960 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.674945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:43591 (local address 127.0.0.1:41122)
0619 06:58:26.675060 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.675074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.675146 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.675338 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.675340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.675456 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.675519 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.675523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.676515 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.676517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.677061 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.677069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.677515 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.677518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.677580 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.678036 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.678049 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.679398 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.680817 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.680820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.680821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.680998 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.681223 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.681225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.681226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.681298 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.681598 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.681600 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.681672 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.681743 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.681829 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:58:26.681993 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.675003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.11.106.124:43591)
0619 06:58:26.675116 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.675120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.675131 (+ 11us) server_negotiation.cc:407] Connection header received
0619 06:58:26.675218 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.675220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.675252 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.675312 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.675606 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.676416 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.677189 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.677414 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.677500 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.679509 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.679512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.679514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.679533 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.680695 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.681078 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.681081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.681082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.681147 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.681371 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.681373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.681374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.681520 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.681591 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.681806 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.681885 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:58:26.684898 18527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: Bootstrap starting.
I20260619 06:58:26.687220 18527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:26.687953 18527 log.cc:824] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:26.689850 18527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: No bootstrap required, opened a new log
I20260619 06:58:26.692780 18527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } }
I20260619 06:58:26.692984 18527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:26.693058 18527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f91ecc0c3184b5282e4146af78161b4, State: Initialized, Role: FOLLOWER
I20260619 06:58:26.693456 18527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } }
I20260619 06:58:26.693907 18533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } } }
I20260619 06:58:26.694065 18533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:58:26.694620 18527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:58:26.698993 18544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:58:26.699084 18544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:58:26.703188 18373 master_runner.cc:428] Detected that this master 6f91ecc0c3184b5282e4146af78161b4 is joining an existing cluster
I20260619 06:58:26.703238 18373 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.106.124:43591
I20260619 06:58:26.714968 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.705526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.11.106.124:43591)
0619 06:58:26.705808 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.705812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.706007 (+ 195us) server_negotiation.cc:407] Connection header received
0619 06:58:26.706741 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.706744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.706784 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.706856 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.707345 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.708098 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.709341 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.709672 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.709754 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.711293 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.711297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.711301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.711329 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.713092 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.713682 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.713685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.713686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.713763 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.714051 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.714054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.714056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.714264 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.714337 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.714679 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.714781 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"spinlock_wait_cycles":768}
I20260619 06:58:26.714980 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.705238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.11.106.125:39157)
0619 06:58:26.705587 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.705592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.706228 (+ 636us) server_negotiation.cc:407] Connection header received
0619 06:58:26.706480 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.706483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.706533 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.706584 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.707296 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.708085 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.709824 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.710160 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.710231 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.711438 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.711442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.711445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.711468 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.713160 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.713730 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.713743 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.713745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.713822 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.714117 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.714120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.714122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.714309 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.714376 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.714687 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.714819 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260619 06:58:26.715435 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.705219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:43591 (local address 127.0.0.1:41134)
0619 06:58:26.705931 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.705944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.706298 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.706935 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.706938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.707132 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.707229 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.707235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.708213 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.708215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.709234 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.709243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.709900 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.709905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.710010 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.710450 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.710465 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.711157 (+ 692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.713205 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.713209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.713214 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.713567 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.713859 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.713864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.713869 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.713942 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.714396 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.714399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.714494 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.715147 (+ 653us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.715268 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":1,"spinlock_wait_cycles":2304,"thread_start_us":92,"threads_started":1}
I20260619 06:58:26.715462 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.705677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.11.106.126:34885)
0619 06:58:26.705834 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.705841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.705856 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:26.705896 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.705898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.705936 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.706020 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.706500 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.707255 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.708613 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.708943 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.709015 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.711066 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.711071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.711076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.711119 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.713694 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.714191 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.714194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.714196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.714273 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.714577 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.714583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.714586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.714790 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.714859 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.715223 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.715324 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:58:26.715873 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.705181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:52612)
0619 06:58:26.705655 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.705668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.705771 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.706040 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.706043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.706259 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.706380 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.706386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.707370 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.707373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.708458 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.708468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.709187 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.709191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.709279 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.709835 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.709854 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.710914 (+ 1060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.713815 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.713820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.713825 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.714074 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.714383 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.714387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.714389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.714469 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.714927 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.714930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.715022 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.715593 (+ 571us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.715717 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"spinlock_wait_cycles":6912,"thread_start_us":95,"threads_started":1}
I20260619 06:58:26.717104 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.705360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.0.0.1:45522)
0619 06:58:26.706158 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.706170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.706381 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.706784 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.706786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.707043 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.707156 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.707166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.708225 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.708228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.709671 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.709684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.710382 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.710386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.710476 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.710832 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.710846 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.711282 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.713280 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.713284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.713286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.713621 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.713928 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.713931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.713933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.714022 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.714446 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.714448 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.714528 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.715877 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.716940 (+ 1063us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":24,"thread_start_us":85,"threads_started":1}
I20260619 06:58:26.718235 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 17871
I20260619 06:58:26.725798 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:36035
--local_ip_for_outbound_sockets=127.11.106.65
--tserver_master_addrs=127.11.106.126:34885
--webserver_port=41981
--webserver_interface=127.11.106.65
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab
--principal=oryx/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.106.126:34885,127.11.106.125:39157
--tserver_master_addrs=127.11.106.126:34885,127.11.106.125:39157,127.11.106.124:43591 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852305, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.126@KRBTEST.COM for oryx/127.11.106.124@KRBTEST.COM
I20260619 06:58:26.749402 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.725276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.11.106.124:43591)
0619 06:58:26.725424 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.725429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.725445 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:26.725532 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.725537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.725603 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.725698 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.726377 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.727190 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.728022 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.728252 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.728369 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.730934 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.730938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.730941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.730967 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.746447 (+ 15480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.747015 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.747018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.747020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.747114 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.747429 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.747432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.747434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.747670 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.748779 (+ 1109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.749085 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.749225 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:58:26.749436 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.719565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:43591 (local address 127.0.0.1:41142)
0619 06:58:26.719702 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.719720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.719804 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.725853 (+ 6049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.725856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.726083 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.726227 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.726239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.727343 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.727345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.727891 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.727898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.728374 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.728378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.728480 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.729070 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.729090 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.730783 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.746614 (+ 15831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.746619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.746621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.746899 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.747229 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.747234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.747236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.747332 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.748787 (+ 1455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.748790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.748898 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.749016 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.749240 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 06:58:26.750918 17809 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.106.124:43591: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4"
member_type: NON_VOTER
last_known_addr {
host: "127.11.106.124"
port: 43591
}
attrs {
promote: true
}
}
dest_uuid: "7bffd54a4b224a75b78337fd20bc4c14"
cas_config_opid_index: 5
I20260619 06:58:26.751540 17809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } }
I20260619 06:58:26.752732 18369 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6f91ecc0c3184b5282e4146af78161b4 (127.11.106.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.754125 18369 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.106.124:43591
I20260619 06:58:26.754302 18369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.754388 18369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:26.755285 18501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:26.756647 18501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 FOLLOWER]: Refusing update from remote peer 7bffd54a4b224a75b78337fd20bc4c14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 06:58:26.757773 18368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:26.757838 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.753009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.11.106.125:39157)
0619 06:58:26.753132 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.753137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.753394 (+ 257us) server_negotiation.cc:407] Connection header received
0619 06:58:26.753580 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.753584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.753648 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.753715 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:26.754080 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.754874 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.756357 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.757033 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.757136 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.757265 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.757643 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.757718 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:58:26.758205 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.752777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.0.0.1:45536)
0619 06:58:26.753333 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.753347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.753464 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.753864 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.753867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.753882 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:26.753965 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.753970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.755025 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.755028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.756220 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.756232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.757394 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.757397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.757485 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.757526 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.758011 (+ 485us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.758083 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489}
I20260619 06:58:26.758702 18419 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 LEARNER]: Refusing update from remote peer 7bffd54a4b224a75b78337fd20bc4c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 06:58:26.759094 18368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 06:58:26.759155 18373 master.cc:537] Master@127.11.106.124:43591 shutting down...
I20260619 06:58:26.761023 18419 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6f91ecc0c3184b5282e4146af78161b4 (127.11.106.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.762996 18448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bffd54a4b224a75b78337fd20bc4c14. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.763103 18448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:58:26.763191 18448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.763271 18448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:58:26.764060 18501 raft_consensus.cc:858] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } }
W20260619 06:58:26.764166 18501 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } }
I20260619 06:58:26.768646 18368 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: attempting to promote NON_VOTER 46bf7f0b36ab4d9a846c19ae6e4991e7 to VOTER
I20260619 06:58:26.768596 18501 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 46bf7f0b36ab4d9a846c19ae6e4991e7 (127.11.106.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } }
I20260619 06:58:26.769088 18368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } }
I20260619 06:58:26.770097 18556 mvcc.cc:205] Tried to move back new op lower bound from 7298467043043545088 to 7298467042920329216. Current Snapshot: MvccSnapshot[applied={T|T < 7298467042957692928}]
I20260619 06:58:26.770771 18533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bffd54a4b224a75b78337fd20bc4c14. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } }
I20260619 06:58:26.770882 18533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 06:58:26.770979 18533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } }
I20260619 06:58:26.771059 18533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 06:58:26.771147 18533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: NON_VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: true } } }
I20260619 06:58:26.771220 18533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 06:58:26.771323 18419 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 LEARNER]: Refusing update from remote peer 7bffd54a4b224a75b78337fd20bc4c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260619 06:58:26.771618 17793 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 -> Peer 6f91ecc0c3184b5282e4146af78161b4 (127.11.106.124:43591): Couldn't send request to peer 6f91ecc0c3184b5282e4146af78161b4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 06:58:26.771718 18369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 06:58:26.773258 18373 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 06:58:26.773499 18373 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 06:58:26.773598 18373 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: stopping tablet replica
I20260619 06:58:26.774905 18369 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 46bf7f0b36ab4d9a846c19ae6e4991e7 (127.11.106.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.776371 18369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.776468 18369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:26.777004 18368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.777100 18368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:26.777675 18419 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 46bf7f0b36ab4d9a846c19ae6e4991e7 (127.11.106.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.779587 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.779703 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:58:26.791729 18373 master.cc:552] Master@127.11.106.124:43591 shutdown complete.
I20260619 06:58:26.791791 18373 master_runner.cc:305] Clearing existing system tablet
I20260619 06:58:26.792035 18373 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:26.792090 18373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:26.793470 18373 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 06:58:26.794394 18566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.794628 18373 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:26.794754 18373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 06:58:26.794787 18373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal
uuid: "6f91ecc0c3184b5282e4146af78161b4"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "9664e9d48a11bd2de9e24e7ee4b05e3d"
server_key_iv: "478f390c7b7508478582a682ed3b8b52"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.796638 18373 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:26.798990 18373 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:58:26.799079 18373 log.cc:1196] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 06:58:26.799371 18373 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: Deleting consensus metadata
I20260619 06:58:26.799530 18373 master_runner.cc:315] Copying system tablet from 127.11.106.126:34885
I20260619 06:58:26.800719 18373 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: tablet copy: Beginning tablet copy session from remote peer at address 127.11.106.126:34885
I20260619 06:58:26.813112 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.801254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:52626)
0619 06:58:26.804723 (+ 3469us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.804740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.804888 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.805196 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.805199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.805434 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.805549 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.805556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.806661 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.806664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.807530 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.807541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.808139 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.808143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.808246 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.808868 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.808883 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.809308 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.811341 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.811345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.811347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.811630 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.811942 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.811945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.811947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.812046 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.812503 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.812505 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.812657 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.812773 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.812926 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3387,"thread_start_us":81,"threads_started":1}
I20260619 06:58:26.813131 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.801356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.11.106.126:34885)
0619 06:58:26.801508 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.801513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.804849 (+ 3336us) server_negotiation.cc:407] Connection header received
0619 06:58:26.805002 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.805005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.805061 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.805166 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.805699 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.806521 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.807669 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.808005 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.808114 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.809463 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.809467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.809470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.809499 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.811223 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.811739 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.811743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.811745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.811836 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.812155 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.812159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.812160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.812386 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.812486 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.812853 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.812977 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:58:26.815114 17829 tablet_copy_service.cc:140] P 7bffd54a4b224a75b78337fd20bc4c14: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6f91ecc0c3184b5282e4146af78161b4 ({username='oryx', principal='oryx/127.11.106.124@KRBTEST.COM'} at 127.0.0.1:52626)
I20260619 06:58:26.815234 17829 tablet_copy_service.cc:161] P 7bffd54a4b224a75b78337fd20bc4c14: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6f91ecc0c3184b5282e4146af78161b4 at {username='oryx', principal='oryx/127.11.106.124@KRBTEST.COM'} at 127.0.0.1:52626: session id = 6f91ecc0c3184b5282e4146af78161b4-00000000000000000000000000000000
I20260619 06:58:26.816372 17829 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:58:26.817644 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:58:26.819998 18373 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: tablet copy: Starting download of 0 data blocks...
I20260619 06:58:26.820180 18373 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: tablet copy: Starting download of 1 WAL segments...
I20260619 06:58:26.821699 18373 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:58:26.823434 17829 tablet_copy_service.cc:342] P 7bffd54a4b224a75b78337fd20bc4c14: Request end of tablet copy session 6f91ecc0c3184b5282e4146af78161b4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.106.124@KRBTEST.COM'} at 127.0.0.1:52626
I20260619 06:58:26.823513 17829 tablet_copy_service.cc:433] P 7bffd54a4b224a75b78337fd20bc4c14: ending tablet copy session 6f91ecc0c3184b5282e4146af78161b4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6f91ecc0c3184b5282e4146af78161b4
I20260619 06:58:26.831813 18373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:26.833547 18576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.833622 18573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:26.834220 18373 server_base.cc:1034] running on GCE node
W20260619 06:58:26.837417 18574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:26.837695 18373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:26.838018 18373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:26.839161 18373 hybrid_clock.cc:648] HybridClock initialized: now 1781852306839146 us; error 33 us; skew 500 ppm
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:26.840816 18373 init.cc:377] Logged in from keytab as oryx/127.11.106.124@KRBTEST.COM (short username oryx)
I20260619 06:58:26.841301 18373 webserver.cc:460] Webserver started at http://127.11.106.124:41567/ using document root <none> and password file <none>
I20260619 06:58:26.841497 18373 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:26.841563 18373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:26.843088 18373 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:26.844172 18583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.844468 18373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:26.844573 18373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal
uuid: "6f91ecc0c3184b5282e4146af78161b4"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "9664e9d48a11bd2de9e24e7ee4b05e3d"
server_key_iv: "478f390c7b7508478582a682ed3b8b52"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.844666 18373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:26.865554 18373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260619 06:58:26.865968 18553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:26.866310 18553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:26.866400 18553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 06:58:26.867767 18373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:26.872483 18373 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.124:43591
I20260619 06:58:26.873273 18373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
W20260619 06:58:26.873425 18553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:26.873492 18553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:26.873592 18553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:26.878484 18639 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:58:26.879554 18639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: Bootstrap starting.
I20260619 06:58:26.879528 18553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab
--principal=oryx/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:36035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=41981
--tserver_master_addrs=127.11.106.126:34885,127.11.106.125:39157,127.11.106.124:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:26.880743 18553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:26.881807 18553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:58:26.882036 18638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.124:43591 every 8 connection(s)
I20260619 06:58:26.887696 18639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:58:26.887981 18639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: Bootstrap complete.
I20260619 06:58:26.888587 18639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } }
I20260619 06:58:26.888739 18639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f91ecc0c3184b5282e4146af78161b4, State: Initialized, Role: LEARNER
I20260619 06:58:26.888888 18639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } }
I20260619 06:58:26.889236 18647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:26.889348 18639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:26.889381 18647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:58:26.892377 18658 catalog_manager.cc:1259] Loaded cluster ID: de7075af517749e9b4831371afb48188
I20260619 06:58:26.892428 18658 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: loading cluster ID for follower catalog manager: success
W20260619 06:58:26.892406 18646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.892488 18644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:26.892406 18643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:26.893277 18134 heartbeater.cc:503] Master 127.11.106.126:34885 was elected leader, sending a full tablet report...
I20260619 06:58:26.893924 18553 server_base.cc:1034] running on GCE node
I20260619 06:58:26.894380 18553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:26.895133 18553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:26.895323 18658 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: acquiring CA information for follower catalog manager: success
I20260619 06:58:26.896333 18553 hybrid_clock.cc:648] HybridClock initialized: now 1781852306896298 us; error 53 us; skew 500 ppm
I20260619 06:58:26.896482 18658 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:26.899082 18553 init.cc:377] Logged in from keytab as oryx/127.11.106.65@KRBTEST.COM (short username oryx)
I20260619 06:58:26.899873 18553 webserver.cc:460] Webserver started at http://127.11.106.65:41981/ using document root <none> and password file <none>
I20260619 06:58:26.900393 18553 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:26.900457 18553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:26.904451 18553 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:26.906492 18665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.907768 18553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:26.907898 18553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "0ea6eac2761c4cc5a736687209d01f3a"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "d1de3622916366519ba3157960648134"
server_key_iv: "b6e2a5815527d7efb10993aae42d7e0a"
server_key_version: "encryptionkey@0"
I20260619 06:58:26.908419 18553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:26.928570 18553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:26.932750 18553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:26.932947 18553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:26.933562 18553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:26.934417 18553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:26.934463 18553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.934500 18553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:26.934515 18553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:26.942970 18553 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:36035
I20260619 06:58:26.943030 18778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:36035 every 8 connection(s)
I20260619 06:58:26.943938 18553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:26.944272 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 18553
I20260619 06:58:26.944487 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 18005
I20260619 06:58:26.953604 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:42889
--local_ip_for_outbound_sockets=127.11.106.66
--tserver_master_addrs=127.11.106.126:34885
--webserver_port=40897
--webserver_interface=127.11.106.66
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab
--principal=oryx/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.106.126:34885,127.11.106.125:39157
--tserver_master_addrs=127.11.106.126:34885,127.11.106.125:39157,127.11.106.124:43591 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.124@KRBTEST.COM
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
Jun 19 06:58:26 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852306, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.125@KRBTEST.COM
I20260619 06:58:26.965090 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.946139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:57061 (local address 127.11.106.126:34885)
0619 06:58:26.951721 (+ 5582us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.951725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.951738 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:58:26.951778 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.951781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.951818 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.951916 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.953466 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.954304 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.955940 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.956290 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.956365 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.960655 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.960659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.960662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.960693 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.962458 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.963134 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.963140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.963145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.963251 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.963672 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.963675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.963677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.963895 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.963978 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.964791 (+ 813us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.964918 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5495}
I20260619 06:58:26.965088 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.949606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:38161 (local address 127.11.106.125:39157)
0619 06:58:26.949765 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.949772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.953498 (+ 3726us) server_negotiation.cc:407] Connection header received
0619 06:58:26.953632 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.953635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.953698 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.953779 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.954590 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.955361 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.957148 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.957479 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.957573 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.960991 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.961001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.961003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.961064 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.962733 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.963216 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.963220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.963221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.963304 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.963671 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.963675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.963677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.963895 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.963978 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.964791 (+ 813us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.964918 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:58:26.966585 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.946540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:43591 (local address 127.11.106.65:34661)
0619 06:58:26.949748 (+ 3208us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.949763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.949884 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.953676 (+ 3792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.953682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.953926 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.954088 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.954099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.955629 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.955635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.956587 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.956595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.957274 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.957278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.957373 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.957837 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.957855 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.959954 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.963369 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.963375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.963379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.963692 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.964059 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.964065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.964069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.964182 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.964744 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.964749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.964865 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.964987 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.965432 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3117,"mutex_wait_us":53,"spinlock_wait_cycles":14080,"thread_start_us":202,"threads_started":1}
I20260619 06:58:26.966867 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.946539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.65:57061)
0619 06:58:26.947201 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.947258 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.948527 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.952117 (+ 3590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.952129 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.952698 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.953313 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.953329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.954466 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.954470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.955810 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.955820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.956599 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.956603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.956814 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.957770 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.957793 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.960463 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.962603 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.962611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.962631 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.963014 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.963371 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.963375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.963376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.963557 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.964050 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.964057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.964453 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.965246 (+ 793us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.965539 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":167,"spinlock_wait_cycles":7680,"thread_start_us":677,"threads_started":1}
I20260619 06:58:26.967443 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.949472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.11.106.65:38161)
0619 06:58:26.953409 (+ 3937us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:26.953422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:26.953523 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:26.954153 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:26.954156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:26.954325 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:26.954459 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.954465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.955500 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.955503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.956381 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:26.956390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.958022 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.958026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:26.958115 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.958959 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:26.958978 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:26.960826 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:26.962849 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.962853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.962855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.963117 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.963404 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:26.963410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:26.963414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:26.963570 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:26.964050 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:26.964057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:26.964453 (+ 396us) client_negotiation.cc:769] Sending connection context
0619 06:58:26.965344 (+ 891us) client_negotiation.cc:240] Negotiation successful
0619 06:58:26.965482 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3860,"mutex_wait_us":48,"spinlock_wait_cycles":6784,"thread_start_us":120,"threads_started":1}
I20260619 06:58:26.967715 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:26.946295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:34661 (local address 127.11.106.124:43591)
0619 06:58:26.952998 (+ 6703us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:26.953010 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:26.953024 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:58:26.953085 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:26.953091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:26.953147 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:26.954216 (+ 1069us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:26.954352 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.955487 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.956714 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:26.957139 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:26.957739 (+ 600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:26.960146 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:26.960162 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:26.960167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:26.960199 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:26.963226 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.963832 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.963838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.963842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.963936 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:26.964314 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:26.964319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:26.964321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:26.964613 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:26.966230 (+ 1617us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:26.966309 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 06:58:26.966435 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6630,"thread_start_us":80,"threads_started":1}
I20260619 06:58:26.968060 18780 heartbeater.cc:347] Connected to a master server at 127.11.106.126:34885
I20260619 06:58:26.968321 18781 heartbeater.cc:347] Connected to a master server at 127.11.106.125:39157
I20260619 06:58:26.968336 18780 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:26.968421 18781 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:26.968997 18780 heartbeater.cc:511] Master 127.11.106.126:34885 requested a full tablet report, sending...
I20260619 06:58:26.969012 18781 heartbeater.cc:511] Master 127.11.106.125:39157 requested a full tablet report, sending...
I20260619 06:58:26.970189 17809 ts_manager.cc:194] Re-registered known tserver with Master: 0ea6eac2761c4cc5a736687209d01f3a (127.11.106.65:36035)
I20260619 06:58:26.970523 18409 ts_manager.cc:194] Registered new tserver with Master: 0ea6eac2761c4cc5a736687209d01f3a (127.11.106.65:36035)
I20260619 06:58:26.970742 17809 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:57061
I20260619 06:58:26.971685 18779 heartbeater.cc:347] Connected to a master server at 127.11.106.124:43591
I20260619 06:58:26.971813 18779 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:26.972043 18779 heartbeater.cc:511] Master 127.11.106.124:43591 requested a full tablet report, sending...
I20260619 06:58:26.973174 18603 ts_manager.cc:194] Registered new tserver with Master: 0ea6eac2761c4cc5a736687209d01f3a (127.11.106.65:36035)
W20260619 06:58:27.074059 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:27.074326 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:27.074368 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:27.078390 18787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:27.078452 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:27.078547 18787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:27.084022 18787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab
--principal=oryx/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:42889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=40897
--tserver_master_addrs=127.11.106.126:34885,127.11.106.125:39157,127.11.106.124:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:27.085232 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:27.086265 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:27.096406 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:27.096484 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:27.096410 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:27.099264 18787 server_base.cc:1034] running on GCE node
I20260619 06:58:27.099643 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:27.100332 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:27.101516 18787 hybrid_clock.cc:648] HybridClock initialized: now 1781852307101475 us; error 50 us; skew 500 ppm
Jun 19 06:58:27 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852307, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:27.104372 18787 init.cc:377] Logged in from keytab as oryx/127.11.106.66@KRBTEST.COM (short username oryx)
I20260619 06:58:27.105239 18787 webserver.cc:460] Webserver started at http://127.11.106.66:40897/ using document root <none> and password file <none>
I20260619 06:58:27.105789 18787 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:27.105895 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:27.109556 18787 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 06:58:27.111541 18802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:27.112902 18787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:27.113021 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "5411bf416f874e9b9acfbaa08c340e20"
format_stamp: "Formatted at 2026-06-19 06:58:25 on dist-test-slave-10d0"
server_key: "b3efd287df7edcb87a62acdcdf2b047e"
server_key_iv: "52dc70a5fb348fa2179be9925d61a6a3"
server_key_version: "encryptionkey@0"
I20260619 06:58:27.113531 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:27.118132 18268 heartbeater.cc:503] Master 127.11.106.126:34885 was elected leader, sending a full tablet report...
I20260619 06:58:27.127238 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:27.130098 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:27.130306 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:27.130951 18787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:27.131824 18787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:27.131893 18787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:27.131969 18787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:27.132012 18787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:27.141136 18787 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:42889
I20260619 06:58:27.141160 18915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:42889 every 8 connection(s)
I20260619 06:58:27.142176 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:27.151343 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 18787
I20260619 06:58:27.151552 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 18139
Jun 19 06:58:27 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852307, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
Jun 19 06:58:27 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852307, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.125@KRBTEST.COM
Jun 19 06:58:27 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852307, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.124@KRBTEST.COM
I20260619 06:58:27.160357 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.144145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:41489 (local address 127.11.106.124:43591)
0619 06:58:27.144824 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.144829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.145648 (+ 819us) server_negotiation.cc:407] Connection header received
0619 06:58:27.146727 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.146730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.146788 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.146842 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.148061 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.148901 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.150937 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.151310 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.151375 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.155585 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.155589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.155592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.155621 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.157950 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158557 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.158561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.158564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.158660 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158984 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.158987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.158989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.159202 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.159269 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.160048 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.160190 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582}
I20260619 06:58:27.160357 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.144324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:49135 (local address 127.11.106.126:34885)
0619 06:58:27.144455 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.144460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.145953 (+ 1493us) server_negotiation.cc:407] Connection header received
0619 06:58:27.146734 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.146737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.146787 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.146842 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.148061 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.148901 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.150936 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.151268 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.151353 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.155585 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.155589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.155592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.155633 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.157567 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158411 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.158415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.158417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.158512 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158917 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.158923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.158927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.159163 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.159245 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.160047 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.160189 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:58:27.160415 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.145025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:59777 (local address 127.11.106.125:39157)
0619 06:58:27.145176 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.145195 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.146008 (+ 813us) server_negotiation.cc:407] Connection header received
0619 06:58:27.146712 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.146716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.146772 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.146854 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.148056 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.148718 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.150907 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.151234 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.151346 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.155348 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.155355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.155362 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.155397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.157764 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158535 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.158538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.158540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.158628 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158953 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.158959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.158964 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.159200 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.159287 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.160048 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.160184 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:58:27.161120 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:39931
--local_ip_for_outbound_sockets=127.11.106.67
--tserver_master_addrs=127.11.106.126:34885
--webserver_port=34933
--webserver_interface=127.11.106.67
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab
--principal=oryx/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.106.126:34885,127.11.106.125:39157
--tserver_master_addrs=127.11.106.126:34885,127.11.106.125:39157,127.11.106.124:43591 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:58:27.162091 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.144684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:43591 (local address 127.11.106.66:41489)
0619 06:58:27.145491 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.145543 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.146493 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.147030 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.147038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.147498 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:27.147902 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.147916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.149085 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.149088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.150796 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.150807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.152656 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.152660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.152744 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.153223 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:27.153236 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:27.155423 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:27.158071 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.158077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.158446 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.158770 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158772 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.158774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.158881 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.159336 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:27.159343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:27.159744 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.160524 (+ 780us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.160990 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":367,"spinlock_wait_cycles":29056,"thread_start_us":146,"threads_started":1}
I20260619 06:58:27.162112 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.144683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.66:49135)
0619 06:58:27.145772 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.145785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.146493 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.147030 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.147038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.147476 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:27.147902 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.147916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.149085 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.149088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.150796 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.150807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.151530 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.151534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.151837 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.152721 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:27.152741 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:27.155423 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:27.157755 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.157762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.157780 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.158268 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.158627 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.158631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.158806 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.159336 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:27.159343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:27.159744 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.161169 (+ 1425us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.161494 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":306,"spinlock_wait_cycles":119296,"thread_start_us":169,"threads_started":1}
I20260619 06:58:27.162405 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.144684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.11.106.66:59777)
0619 06:58:27.145908 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.145924 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.146499 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.147029 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.147036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.147418 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:27.147900 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.147914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.150097 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.150100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.150780 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.150789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.151530 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.151534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.152149 (+ 615us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.152827 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:27.152848 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:27.155132 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:27.157950 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.157956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.157962 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.158417 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.158741 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.158746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.158751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.158856 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.159363 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:27.159369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:27.159731 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.160592 (+ 861us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.161083 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"mutex_wait_us":640,"spinlock_wait_cycles":14464,"thread_start_us":651,"threads_started":1}
I20260619 06:58:27.163429 18917 heartbeater.cc:347] Connected to a master server at 127.11.106.126:34885
I20260619 06:58:27.163479 18916 heartbeater.cc:347] Connected to a master server at 127.11.106.124:43591
I20260619 06:58:27.163672 18917 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:27.163672 18916 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:27.164202 18916 heartbeater.cc:511] Master 127.11.106.124:43591 requested a full tablet report, sending...
I20260619 06:58:27.164199 18917 heartbeater.cc:511] Master 127.11.106.126:34885 requested a full tablet report, sending...
I20260619 06:58:27.163722 18918 heartbeater.cc:347] Connected to a master server at 127.11.106.125:39157
I20260619 06:58:27.164695 18918 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:27.164848 18918 heartbeater.cc:511] Master 127.11.106.125:39157 requested a full tablet report, sending...
I20260619 06:58:27.165433 17809 ts_manager.cc:194] Re-registered known tserver with Master: 5411bf416f874e9b9acfbaa08c340e20 (127.11.106.66:42889)
I20260619 06:58:27.165522 18409 ts_manager.cc:194] Registered new tserver with Master: 5411bf416f874e9b9acfbaa08c340e20 (127.11.106.66:42889)
I20260619 06:58:27.165524 18603 ts_manager.cc:194] Registered new tserver with Master: 5411bf416f874e9b9acfbaa08c340e20 (127.11.106.66:42889)
I20260619 06:58:27.166282 17809 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:49135
I20260619 06:58:27.217902 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.213941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.11.106.124:43591)
0619 06:58:27.214102 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.214106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.214122 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:27.214257 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.214260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.214319 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.214412 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:27.214757 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.215284 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.216747 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.217364 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.217475 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.217710 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.217743 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.217800 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:58:27.217907 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.213846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:43591 (local address 127.0.0.1:41158)
0619 06:58:27.214008 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.214026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.214134 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.214451 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.214456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.214483 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:27.214613 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.214622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.215400 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.215404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.216594 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.216606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.217490 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.217494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.217585 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.217613 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.217684 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.217765 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 06:58:27.218628 18605 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 06:58:27.218918 18647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bffd54a4b224a75b78337fd20bc4c14. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:27.219069 18647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: This master's current role is: LEARNER
W20260619 06:58:27.277428 18923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:27.277706 18923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:27.277791 18923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:27.282160 18923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:27.282212 18923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:27.282306 18923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:27.287660 18923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab
--principal=oryx/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:39931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=34933
--tserver_master_addrs=127.11.106.126:34885,127.11.106.125:39157,127.11.106.124:43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:27.288969 18923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:27.290059 18923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:27.299643 18929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:27.299643 18928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:27.299805 18923 server_base.cc:1034] running on GCE node
W20260619 06:58:27.299963 18931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:27.300343 18923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:27.300997 18923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:27.302152 18923 hybrid_clock.cc:648] HybridClock initialized: now 1781852307302136 us; error 48 us; skew 500 ppm
Jun 19 06:58:27 dist-test-slave-10d0 krb5kdc[17764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852307, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:27.304975 18923 init.cc:377] Logged in from keytab as oryx/127.11.106.67@KRBTEST.COM (short username oryx)
I20260619 06:58:27.305832 18923 webserver.cc:460] Webserver started at http://127.11.106.67:34933/ using document root <none> and password file <none>
I20260619 06:58:27.306339 18923 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:27.306406 18923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:27.310209 18923 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:58:27.312328 18938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:27.313805 18923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:27.313935 18923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "7c0af17d04af45ca901f90a9ade63781"
format_stamp: "Formatted at 2026-06-19 06:58:26 on dist-test-slave-10d0"
server_key: "42560fb8d942a6f5da20ea468d72e423"
server_key_iv: "90a50f7b27a1e6b2e2ef0c99c6a06b21"
server_key_version: "encryptionkey@0"
I20260619 06:58:27.314482 18923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:27.332191 18923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:27.335436 18923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:27.335631 18923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:27.336293 18923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:27.337231 18923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:27.337278 18923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:27.337348 18923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:27.337378 18923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:27.346098 18923 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:39931
I20260619 06:58:27.346127 19051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:39931 every 8 connection(s)
I20260619 06:58:27.347074 18923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:27.349979 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 18923
Jun 19 06:58:27 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852307, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.124@KRBTEST.COM
Jun 19 06:58:27 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852307, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:27.363633 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.352310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.11.106.124:43591)
0619 06:58:27.353034 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.353048 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.353064 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:27.353480 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.353483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.353531 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.353633 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.354739 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.355489 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.356627 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.357031 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.357126 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.359090 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.359097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.359100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.359155 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.361476 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.361966 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.361970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.361972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.362071 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.362383 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.362386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.362388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.362599 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.362735 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.363303 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.363423 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":970,"server-negotiator.queue_time_us":645,"thread_start_us":83,"threads_started":1}
I20260619 06:58:27.365897 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.349279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:36337 (local address 127.11.106.126:34885)
0619 06:58:27.349525 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.349529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.350925 (+ 1396us) server_negotiation.cc:407] Connection header received
0619 06:58:27.351664 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.351668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.351724 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.351809 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.353663 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.354484 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.356104 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.356452 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.356567 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.360335 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.360339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.360342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.360366 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.362187 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.362818 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.362824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.362829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.362923 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.364242 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.364247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.364249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.364476 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.364583 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.365331 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.365717 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260619 06:58:27.366726 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.349585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:45903 (local address 127.11.106.124:43591)
0619 06:58:27.350462 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.350468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.350757 (+ 289us) server_negotiation.cc:407] Connection header received
0619 06:58:27.351664 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.351668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.351767 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.351838 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.353663 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.354484 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.356581 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.356897 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.356990 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.360871 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.360876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.360880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.360908 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.362784 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.364992 (+ 2208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.364995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.364997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.365086 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.365415 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.365419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.365422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.365663 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.365759 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.366444 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.366554 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":790}
I20260619 06:58:27.367573 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.349404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:43591 (local address 127.11.106.67:45903)
0619 06:58:27.350636 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.350667 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.351348 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.352033 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.352042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.352759 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:27.353475 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.353494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.354635 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.354639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.356431 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.356440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.357102 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.357109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.357221 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.357741 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:27.357753 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:27.360184 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:27.362894 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.362897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.362899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.364890 (+ 1991us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.365205 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.365208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.365212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.365298 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.365808 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:27.365813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:27.365910 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.366315 (+ 405us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.367252 (+ 937us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"mutex_wait_us":1920,"spinlock_wait_cycles":2304,"thread_start_us":152,"threads_started":1}
I20260619 06:58:27.367686 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.352767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.11.106.126:34885)
0619 06:58:27.353365 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.353370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.353943 (+ 573us) server_negotiation.cc:407] Connection header received
0619 06:58:27.354129 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.354134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.354210 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.354284 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.355218 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.356075 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.358153 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.358609 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.358698 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.362667 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.362671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.362674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.362697 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.364739 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.365361 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.365363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.365365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.365447 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.365788 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.365792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.365794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.365991 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.366105 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.367395 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.367515 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":84,"threads_started":1}
I20260619 06:58:27.367573 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.349404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.67:36337)
0619 06:58:27.350838 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.350851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.351348 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.352031 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.352041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.352668 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:27.353475 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.353494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.354635 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.354639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.355972 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.355983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.356717 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.356720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.356905 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.357576 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:27.357598 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:27.359989 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:27.362337 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.362343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.362358 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.362696 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.363028 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.363033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.363038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.363879 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.364678 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:27.364685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:27.365012 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.366108 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.366507 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"mutex_wait_us":1130,"spinlock_wait_cycles":1920,"thread_start_us":82,"threads_started":1}
I20260619 06:58:27.368891 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.352927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.11.106.125:39157)
0619 06:58:27.353228 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.353234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.354860 (+ 1626us) server_negotiation.cc:407] Connection header received
0619 06:58:27.355031 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.355036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.355082 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.355145 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.360634 (+ 5489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.361489 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.363026 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.363357 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.363432 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.365145 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.365149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.365152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.365181 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.366937 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.367415 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.367419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.367421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.367502 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.367857 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.367860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.367862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.368080 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.368183 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.368567 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.368710 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":71,"threads_started":1}
I20260619 06:58:27.369306 19052 heartbeater.cc:347] Connected to a master server at 127.11.106.124:43591
I20260619 06:58:27.369413 19053 heartbeater.cc:347] Connected to a master server at 127.11.106.126:34885
I20260619 06:58:27.369611 19053 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:27.369702 19052 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:27.370186 19053 heartbeater.cc:511] Master 127.11.106.126:34885 requested a full tablet report, sending...
I20260619 06:58:27.370216 19052 heartbeater.cc:511] Master 127.11.106.124:43591 requested a full tablet report, sending...
I20260619 06:58:27.371560 17808 ts_manager.cc:194] Re-registered known tserver with Master: 7c0af17d04af45ca901f90a9ade63781 (127.11.106.67:39931)
I20260619 06:58:27.372236 17808 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:36337
I20260619 06:58:27.372866 18603 ts_manager.cc:194] Registered new tserver with Master: 7c0af17d04af45ca901f90a9ade63781 (127.11.106.67:39931)
Jun 19 06:58:27 dist-test-slave-10d0 krb5kdc[17764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852307, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.125@KRBTEST.COM
I20260619 06:58:27.371383 17809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:27.375717 17809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:27.377547 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.349404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.11.106.67:55661)
0619 06:58:27.350152 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.350206 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.351335 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.352031 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.352041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.357532 (+ 5491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:27.357680 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.359814 (+ 2134us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.366413 (+ 6599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.366416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.367195 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.367205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.371054 (+ 3849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.371060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.371158 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.372195 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:27.372206 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:27.373982 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:27.375905 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.375909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.375911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.376151 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.376489 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:27.376492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:27.376494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:27.376612 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:27.377053 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:27.377056 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:27.377134 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.377223 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.377367 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":149,"threads_started":1}
I20260619 06:58:27.377578 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.349133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:55661 (local address 127.11.106.125:39157)
0619 06:58:27.349428 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.349433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.350301 (+ 868us) server_negotiation.cc:407] Connection header received
0619 06:58:27.351584 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.351588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.351646 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.351705 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.359956 (+ 8251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.360743 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.368437 (+ 7694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.368762 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.368840 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.374148 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.374152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.374155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.374181 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.375804 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.376290 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.376294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.376296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.376386 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.376730 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.376734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.376736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.376928 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.377001 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.377326 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.377442 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260619 06:58:27.378384 19054 heartbeater.cc:347] Connected to a master server at 127.11.106.125:39157
I20260619 06:58:27.378475 19054 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:27.378674 19054 heartbeater.cc:511] Master 127.11.106.125:39157 requested a full tablet report, sending...
I20260619 06:58:27.379433 19071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: NON_VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: true } } }
I20260619 06:58:27.379558 19071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:27.381201 19070 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: attempting to promote NON_VOTER 6f91ecc0c3184b5282e4146af78161b4 to VOTER
I20260619 06:58:27.381554 19070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } }
I20260619 06:58:27.381772 18409 ts_manager.cc:194] Registered new tserver with Master: 7c0af17d04af45ca901f90a9ade63781 (127.11.106.67:39931)
I20260619 06:58:27.382725 18419 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Refusing update from remote peer 7bffd54a4b224a75b78337fd20bc4c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 06:58:27.383000 18605 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 LEARNER]: Refusing update from remote peer 7bffd54a4b224a75b78337fd20bc4c14: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 06:58:27.383083 19071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:58:27.383471 19071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:58:27.384814 19072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6f91ecc0c3184b5282e4146af78161b4 (127.11.106.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:27.385221 18419 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6f91ecc0c3184b5282e4146af78161b4 (127.11.106.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:27.387071 19079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:27.387264 19079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:58:27.387216 19071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:27.387310 19071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:27.387351 18605 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6f91ecc0c3184b5282e4146af78161b4 (127.11.106.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:27.387502 19072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:27.387535 19071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:27.387593 19072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:27.387599 19071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:27.388828 18647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7bffd54a4b224a75b78337fd20bc4c14" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:27.388935 18647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:58:27.400667 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.395570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:36035 (local address 127.0.0.1:47350)
0619 06:58:27.395779 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.395791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.395870 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.396751 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.396754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.396776 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:27.396914 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.396921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.398036 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.398042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.399123 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.399131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.400199 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.400203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.400300 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.400347 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.400423 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.400491 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":33,"thread_start_us":85,"threads_started":1}
I20260619 06:58:27.400799 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.395824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.11.106.65:36035)
0619 06:58:27.396128 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.396140 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.396170 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:58:27.396292 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.396301 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.396578 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.397073 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:27.397209 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.397919 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.399248 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.400077 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.400188 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.400336 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.400454 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.400576 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":120,"threads_started":1}
I20260619 06:58:27.401079 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.395236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:39931 (local address 127.0.0.1:55502)
0619 06:58:27.395474 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.395486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.395562 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.396878 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.396881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.396895 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:27.396986 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.396995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.398444 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.398448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.399619 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.399629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.400690 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.400694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.400781 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.400812 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.400866 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.400942 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":5,"thread_start_us":110,"threads_started":1}
I20260619 06:58:27.401300 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.394973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:42889 (local address 127.0.0.1:60166)
0619 06:58:27.395116 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.395134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.395232 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.397494 (+ 2262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.397498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.397515 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:27.397589 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.397594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.399084 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.399088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.399850 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.399859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.400924 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.400927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.401009 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.401028 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.401113 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.401178 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260619 06:58:27.401358 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.395164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.11.106.66:42889)
0619 06:58:27.397000 (+ 1836us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.397013 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.397034 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:58:27.397108 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.397112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.397301 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.397743 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:27.397871 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.398948 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.399974 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.400806 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.400916 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.401046 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.401138 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.401224 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1734,"thread_start_us":104,"threads_started":1}
I20260619 06:58:27.402086 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.395944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.11.106.67:39931)
0619 06:58:27.396366 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.396379 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.396402 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:58:27.396479 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.396484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.396725 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.396897 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:27.397088 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.398301 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.399781 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.400560 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.401692 (+ 1132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.401831 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.401877 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.401956 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":86,"threads_started":1}
I20260619 06:58:27.404160 18713 tablet_service.cc:1467] Processing CreateTablet for tablet e1cf676aa4ca4a2190d93b5ff2459c7d (DEFAULT_TABLE table=test-table [id=9dfa9d619ab641dcb2599c53d48f7580]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:27.404325 18850 tablet_service.cc:1467] Processing CreateTablet for tablet e1cf676aa4ca4a2190d93b5ff2459c7d (DEFAULT_TABLE table=test-table [id=9dfa9d619ab641dcb2599c53d48f7580]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:27.404371 18986 tablet_service.cc:1467] Processing CreateTablet for tablet e1cf676aa4ca4a2190d93b5ff2459c7d (DEFAULT_TABLE table=test-table [id=9dfa9d619ab641dcb2599c53d48f7580]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:27.405171 18713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cf676aa4ca4a2190d93b5ff2459c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:27.405367 18986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cf676aa4ca4a2190d93b5ff2459c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:27.405367 18850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cf676aa4ca4a2190d93b5ff2459c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:27.411976 19086 tablet_bootstrap.cc:492] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: Bootstrap starting.
I20260619 06:58:27.413076 19087 tablet_bootstrap.cc:492] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: Bootstrap starting.
I20260619 06:58:27.414860 19086 tablet_bootstrap.cc:654] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:27.415095 19087 tablet_bootstrap.cc:654] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:27.415763 19086 log.cc:824] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:27.415818 19087 log.cc:824] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:27.416435 19088 tablet_bootstrap.cc:492] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: Bootstrap starting.
I20260619 06:58:27.417886 19087 tablet_bootstrap.cc:492] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: No bootstrap required, opened a new log
I20260619 06:58:27.418079 19086 tablet_bootstrap.cc:492] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: No bootstrap required, opened a new log
I20260619 06:58:27.418126 19087 ts_tablet_manager.cc:1397] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:58:27.418255 19086 ts_tablet_manager.cc:1397] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:58:27.418653 19088 tablet_bootstrap.cc:654] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:27.419443 19088 log.cc:824] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:27.421515 19088 tablet_bootstrap.cc:492] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: No bootstrap required, opened a new log
I20260619 06:58:27.421702 19088 ts_tablet_manager.cc:1397] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:58:27.421816 19086 raft_consensus.cc:348] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } }
I20260619 06:58:27.422055 19086 raft_consensus.cc:374] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:27.422117 19086 raft_consensus.cc:729] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea6eac2761c4cc5a736687209d01f3a, State: Initialized, Role: FOLLOWER
I20260619 06:58:27.422562 19086 consensus_queue.cc:260] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } }
I20260619 06:58:27.422691 19087 raft_consensus.cc:348] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } }
I20260619 06:58:27.422924 19087 raft_consensus.cc:374] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:27.422976 19087 raft_consensus.cc:729] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0af17d04af45ca901f90a9ade63781, State: Initialized, Role: FOLLOWER
I20260619 06:58:27.423384 19087 consensus_queue.cc:260] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } }
I20260619 06:58:27.423486 19086 ts_tablet_manager.cc:1428] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 06:58:27.423885 18780 heartbeater.cc:503] Master 127.11.106.126:34885 was elected leader, sending a full tablet report...
I20260619 06:58:27.423961 19053 heartbeater.cc:503] Master 127.11.106.126:34885 was elected leader, sending a full tablet report...
I20260619 06:58:27.424291 19087 ts_tablet_manager.cc:1428] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 06:58:27.430444 19088 raft_consensus.cc:348] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } }
I20260619 06:58:27.430649 19088 raft_consensus.cc:374] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:27.430743 19088 raft_consensus.cc:729] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5411bf416f874e9b9acfbaa08c340e20, State: Initialized, Role: FOLLOWER
I20260619 06:58:27.431116 19088 consensus_queue.cc:260] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } }
I20260619 06:58:27.431741 18917 heartbeater.cc:503] Master 127.11.106.126:34885 was elected leader, sending a full tablet report...
I20260619 06:58:27.432046 19088 ts_tablet_manager.cc:1428] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
W20260619 06:58:27.445706 18782 tablet.cc:2369] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:58:27.599072 19055 tablet.cc:2369] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:27.616037 19093 raft_consensus.cc:482] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:27.616302 19093 raft_consensus.cc:504] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } }
I20260619 06:58:27.617693 19093 leader_election.cc:288] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5411bf416f874e9b9acfbaa08c340e20 (127.11.106.66:42889), 0ea6eac2761c4cc5a736687209d01f3a (127.11.106.65:36035)
I20260619 06:58:27.621670 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.618270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:36035 (local address 127.11.106.67:47959)
0619 06:58:27.618403 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.618417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.618495 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.618839 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.618842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.618864 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:27.618970 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.618977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.619759 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.619766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.620591 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.620601 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.621230 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.621233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.621339 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.621360 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.621457 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.621524 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 06:58:27.621655 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.618462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:47959 (local address 127.11.106.65:36035)
0619 06:58:27.618587 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.618591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.618603 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:58:27.618651 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.618653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.618741 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.618819 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:27.619097 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.619653 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.620723 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.621111 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.621241 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.621304 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.621473 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.621541 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:58:27.621717 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.618010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:51179 (local address 127.11.106.66:42889)
0619 06:58:27.618145 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.618152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.618173 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:58:27.618595 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.618597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.618635 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.618689 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:27.619002 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.619691 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.620777 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.621184 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.621312 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.621363 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.621532 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.621603 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 06:58:27.621713 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.617941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:42889 (local address 127.11.106.67:51179)
0619 06:58:27.618079 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:27.618096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:27.618249 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:27.618732 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:27.618736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:27.618752 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:27.618889 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.618898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.619825 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.619828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.620628 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:27.620638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.621304 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.621308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:27.621403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.621430 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:58:27.621518 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:58:27.621585 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 06:58:27.622362 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cf676aa4ca4a2190d93b5ff2459c7d" candidate_uuid: "7c0af17d04af45ca901f90a9ade63781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6eac2761c4cc5a736687209d01f3a" is_pre_election: true
I20260619 06:58:27.622502 18870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cf676aa4ca4a2190d93b5ff2459c7d" candidate_uuid: "7c0af17d04af45ca901f90a9ade63781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5411bf416f874e9b9acfbaa08c340e20" is_pre_election: true
I20260619 06:58:27.622692 18733 raft_consensus.cc:2454] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0af17d04af45ca901f90a9ade63781 in term 0.
I20260619 06:58:27.622725 18870 raft_consensus.cc:2454] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c0af17d04af45ca901f90a9ade63781 in term 0.
I20260619 06:58:27.623174 18939 leader_election.cc:302] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6eac2761c4cc5a736687209d01f3a, 7c0af17d04af45ca901f90a9ade63781; no voters:
I20260619 06:58:27.623431 19093 raft_consensus.cc:2781] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:27.623517 19093 raft_consensus.cc:482] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:27.623579 19093 raft_consensus.cc:3037] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:27.624799 19093 raft_consensus.cc:504] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } }
I20260619 06:58:27.625165 19093 leader_election.cc:288] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [CANDIDATE]: Term 1 election: Requested vote from peers 5411bf416f874e9b9acfbaa08c340e20 (127.11.106.66:42889), 0ea6eac2761c4cc5a736687209d01f3a (127.11.106.65:36035)
I20260619 06:58:27.625527 18870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cf676aa4ca4a2190d93b5ff2459c7d" candidate_uuid: "7c0af17d04af45ca901f90a9ade63781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5411bf416f874e9b9acfbaa08c340e20"
I20260619 06:58:27.625555 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cf676aa4ca4a2190d93b5ff2459c7d" candidate_uuid: "7c0af17d04af45ca901f90a9ade63781" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6eac2761c4cc5a736687209d01f3a"
I20260619 06:58:27.625655 18870 raft_consensus.cc:3037] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:27.625660 18733 raft_consensus.cc:3037] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:27.626899 18870 raft_consensus.cc:2454] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0af17d04af45ca901f90a9ade63781 in term 1.
I20260619 06:58:27.626899 18733 raft_consensus.cc:2454] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c0af17d04af45ca901f90a9ade63781 in term 1.
I20260619 06:58:27.627182 18939 leader_election.cc:302] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6eac2761c4cc5a736687209d01f3a, 7c0af17d04af45ca901f90a9ade63781; no voters:
I20260619 06:58:27.627334 19093 raft_consensus.cc:2781] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:27.627562 19093 raft_consensus.cc:686] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 1 LEADER]: Becoming Leader. State: Replica: 7c0af17d04af45ca901f90a9ade63781, State: Running, Role: LEADER
I20260619 06:58:27.627931 19093 consensus_queue.cc:237] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } }
I20260619 06:58:27.631481 17807 catalog_manager.cc:5495] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c0af17d04af45ca901f90a9ade63781 (127.11.106.67). New cstate: current_term: 1 leader_uuid: "7c0af17d04af45ca901f90a9ade63781" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } health_report { overall_health: UNKNOWN } } }
W20260619 06:58:27.644428 18919 tablet.cc:2369] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:27.698001 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.694385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.11.106.67:39931)
0619 06:58:27.694520 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.694525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.694540 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:27.694701 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.694705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.694773 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.694893 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:27.695128 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.695639 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.696518 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.696771 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.696887 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.697207 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:27.697624 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:27.697765 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.697801 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.697856 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:58:27.704015 18870 raft_consensus.cc:1261] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [term 1 FOLLOWER]: Refusing update from remote peer 7c0af17d04af45ca901f90a9ade63781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:27.704015 18733 raft_consensus.cc:1261] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [term 1 FOLLOWER]: Refusing update from remote peer 7c0af17d04af45ca901f90a9ade63781: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:27.704838 19093 consensus_queue.cc:1035] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:27.705027 19095 consensus_queue.cc:1035] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:27.716339 19099 mvcc.cc:205] Tried to move back new op lower bound from 7298467052348010496 to 7298467052047323136. Current Snapshot: MvccSnapshot[applied={T|T < 7298467052348010496}]
I20260619 06:58:27.719348 19100 mvcc.cc:205] Tried to move back new op lower bound from 7298467052348010496 to 7298467052047323136. Current Snapshot: MvccSnapshot[applied={T|T < 7298467052348010496}]
I20260619 06:58:27.723407 17807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:27.723582 17807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:27.727936 17807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bffd54a4b224a75b78337fd20bc4c14: Sending DeleteTablet for 3 replicas of tablet e1cf676aa4ca4a2190d93b5ff2459c7d
I20260619 06:58:27.728947 18850 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf676aa4ca4a2190d93b5ff2459c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:27 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:60166
I20260619 06:58:27.729395 18713 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf676aa4ca4a2190d93b5ff2459c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:27 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:47350
I20260619 06:58:27.730013 19108 tablet_replica.cc:330] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: stopping tablet replica
I20260619 06:58:27.730461 18986 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf676aa4ca4a2190d93b5ff2459c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:27 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:55502
I20260619 06:58:27.730962 19107 tablet_replica.cc:330] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: stopping tablet replica
I20260619 06:58:27.731389 19107 raft_consensus.cc:2229] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:27.731747 19107 raft_consensus.cc:2258] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:27.730448 19108 raft_consensus.cc:2229] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:27.732792 19108 raft_consensus.cc:2258] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:27.732914 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 17777
I20260619 06:58:27.733358 19107 ts_tablet_manager.cc:1905] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:27.734769 19108 ts_tablet_manager.cc:1905] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:27.736138 19107 ts_tablet_manager.cc:1918] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:27.736258 19107 log.cc:1196] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/e1cf676aa4ca4a2190d93b5ff2459c7d
I20260619 06:58:27.737129 19108 ts_tablet_manager.cc:1918] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:27.737241 19108 log.cc:1196] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/e1cf676aa4ca4a2190d93b5ff2459c7d
I20260619 06:58:27.737525 19108 ts_tablet_manager.cc:1939] T e1cf676aa4ca4a2190d93b5ff2459c7d P 0ea6eac2761c4cc5a736687209d01f3a: Deleting consensus metadata
I20260619 06:58:27.740727 19107 ts_tablet_manager.cc:1939] T e1cf676aa4ca4a2190d93b5ff2459c7d P 5411bf416f874e9b9acfbaa08c340e20: Deleting consensus metadata
I20260619 06:58:27.742147 19109 tablet_replica.cc:330] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: stopping tablet replica
I20260619 06:58:27.742607 19109 raft_consensus.cc:2229] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:27.743201 19109 raft_consensus.cc:2258] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 06:58:27.743624 18669 connection.cc:535] server connection from 127.0.0.1:47350 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 06:58:27.744249 18803 connection.cc:535] server connection from 127.0.0.1:60166 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 06:58:27.745281 19109 ts_tablet_manager.cc:1905] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:27.747211 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.732091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.11.106.125:39157)
0619 06:58:27.732329 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.732334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.733254 (+ 920us) server_negotiation.cc:407] Connection header received
0619 06:58:27.733529 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.733536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.733595 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.733661 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.739377 (+ 5716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.739915 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.740940 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.741267 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.741350 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.743301 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.743306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.743311 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.743340 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.745151 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.745724 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.745728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.745730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.745825 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.746161 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.746164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.746166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.746384 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.746502 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.746866 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.746990 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260619 06:58:27.748605 19109 ts_tablet_manager.cc:1918] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:27.748775 19109 log.cc:1196] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/e1cf676aa4ca4a2190d93b5ff2459c7d
I20260619 06:58:27.749184 19109 ts_tablet_manager.cc:1939] T e1cf676aa4ca4a2190d93b5ff2459c7d P 7c0af17d04af45ca901f90a9ade63781: Deleting consensus metadata
W20260619 06:58:27.750449 18941 connection.cc:441] server connection from 127.0.0.1:55502 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55502 (request call id 1) could send its response
I20260619 06:58:27.757899 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.732207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.11.106.124:43591)
0619 06:58:27.732418 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.732425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.748598 (+ 16173us) server_negotiation.cc:407] Connection header received
0619 06:58:27.748814 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.748819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.748898 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.748961 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.749920 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.750673 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.751530 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.751864 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.751981 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.753737 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.753741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.753743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.753768 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.755737 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.756222 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.756227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.756229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.756317 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.756841 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.756848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.756851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.757094 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.757206 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.757561 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.757696 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260619 06:58:27.759642 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.748111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.11.106.125:39157)
0619 06:58:27.748225 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.748229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.749974 (+ 1745us) server_negotiation.cc:407] Connection header received
0619 06:58:27.750127 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.750130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.750182 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.750243 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.751674 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.752437 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.753577 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.753914 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.753981 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.755554 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.755558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.755561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.755587 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.757577 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.758361 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.758365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.758367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.758446 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.758741 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.758744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.758746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.758939 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.759030 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.759380 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.759491 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:58:27.761005 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:27.749395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.11.106.124:43591)
0619 06:58:27.749528 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:27.749532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:27.752009 (+ 2477us) server_negotiation.cc:407] Connection header received
0619 06:58:27.752186 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:27.752190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:27.752230 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:27.752281 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:27.753321 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.754068 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.755418 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:27.755796 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:27.755863 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:27.757430 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:27.757435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:27.757441 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:27.757471 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:27.759140 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.759577 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.759582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.759583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.759771 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:27.760095 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:27.760099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:27.760101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:27.760301 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:27.760423 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:27.760744 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 06:58:27.760860 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 06:58:28.720139 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:28.719672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.65:38793)
0619 06:58:28.719963 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:28.720046 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
W20260619 06:58:28.720638 18780 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:34885 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
I20260619 06:58:28.723189 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:28.722518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.66:51549)
0619 06:58:28.722940 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:28.723057 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":99,"threads_started":1}
W20260619 06:58:28.723532 18917 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:34885 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
I20260619 06:58:28.723661 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:28.723250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.67:41405)
0619 06:58:28.723465 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:28.723554 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
W20260619 06:58:28.723910 19053 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:34885 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
I20260619 06:58:29.228523 19121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7bffd54a4b224a75b78337fd20bc4c14)
I20260619 06:58:29.228826 19121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } }
I20260619 06:58:29.229313 19122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7bffd54a4b224a75b78337fd20bc4c14)
I20260619 06:58:29.229516 19122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } }
I20260619 06:58:29.229981 19121 leader_election.cc:288] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7bffd54a4b224a75b78337fd20bc4c14 (127.11.106.126:34885), 46bf7f0b36ab4d9a846c19ae6e4991e7 (127.11.106.125:39157)
I20260619 06:58:29.230611 19122 leader_election.cc:288] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7bffd54a4b224a75b78337fd20bc4c14 (127.11.106.126:34885), 6f91ecc0c3184b5282e4146af78161b4 (127.11.106.124:43591)
I20260619 06:58:29.231031 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.230105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:53106)
0619 06:58:29.230469 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.230555 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
I20260619 06:58:29.231174 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.230729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:53112)
0619 06:58:29.230926 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.230994 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
W20260619 06:58:29.231227 18585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
W20260619 06:58:29.231360 18394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
I20260619 06:58:29.232189 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.231928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:53114)
0619 06:58:29.232068 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.232117 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 06:58:29.232332 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.232034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:53128)
0619 06:58:29.232151 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.232190 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 06:58:29.232331 18585 leader_election.cc:334] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7bffd54a4b224a75b78337fd20bc4c14 (127.11.106.126:34885): Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
W20260619 06:58:29.232483 18394 leader_election.cc:334] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7bffd54a4b224a75b78337fd20bc4c14 (127.11.106.126:34885): Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
I20260619 06:58:29.234980 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.230941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.11.106.124:43591)
0619 06:58:29.231467 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:29.231471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:29.231483 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:58:29.231524 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:29.231526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:29.231578 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:29.231921 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:29.232005 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.232865 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.233961 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.234466 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.234565 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.234602 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:29.234829 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 06:58:29.234889 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":449,"thread_start_us":51,"threads_started":1}
I20260619 06:58:29.234993 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.230797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:43591 (local address 127.0.0.1:41196)
0619 06:58:29.231158 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.231171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:29.231256 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:29.231703 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:29.231706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:29.231721 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:29.231806 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.231817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.232986 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.232989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.233824 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.233835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.234592 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.234596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.234688 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.234712 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:58:29.234803 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:58:29.234873 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":59,"threads_started":1}
I20260619 06:58:29.235463 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f91ecc0c3184b5282e4146af78161b4" is_pre_election: true
I20260619 06:58:29.235671 18605 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46bf7f0b36ab4d9a846c19ae6e4991e7 in term 1.
I20260619 06:58:29.236086 18393 leader_election.cc:302] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46bf7f0b36ab4d9a846c19ae6e4991e7, 6f91ecc0c3184b5282e4146af78161b4; no voters: 7bffd54a4b224a75b78337fd20bc4c14
I20260619 06:58:29.236255 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.230241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:39157 (local address 127.0.0.1:45568)
0619 06:58:29.231025 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.231042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:29.231204 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:29.231622 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:29.231628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:29.231651 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:29.231766 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.231774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.233961 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.233963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.235008 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.235021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.235869 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.235874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.235973 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.235996 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:58:29.236064 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:58:29.236115 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":31,"thread_start_us":71,"threads_started":1}
I20260619 06:58:29.236289 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.230326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.11.106.125:39157)
0619 06:58:29.230579 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:29.230584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:29.231114 (+ 530us) server_negotiation.cc:407] Connection header received
0619 06:58:29.231342 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:29.231348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:29.231394 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:29.231454 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:29.231897 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.232865 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.235150 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.235732 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.235849 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.235902 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:29.236102 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 06:58:29.236178 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260619 06:58:29.236322 19122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 06:58:29.236383 19122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7bffd54a4b224a75b78337fd20bc4c14)
I20260619 06:58:29.236431 19122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:58:29.236992 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f91ecc0c3184b5282e4146af78161b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" is_pre_election: true
I20260619 06:58:29.237671 19122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } }
I20260619 06:58:29.237927 18419 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f91ecc0c3184b5282e4146af78161b4 in current term 2: Already voted for candidate 46bf7f0b36ab4d9a846c19ae6e4991e7 in this term.
I20260619 06:58:29.238068 19122 leader_election.cc:288] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [CANDIDATE]: Term 2 election: Requested vote from peers 7bffd54a4b224a75b78337fd20bc4c14 (127.11.106.126:34885), 6f91ecc0c3184b5282e4146af78161b4 (127.11.106.124:43591)
I20260619 06:58:29.238341 18586 leader_election.cc:302] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f91ecc0c3184b5282e4146af78161b4; no voters: 46bf7f0b36ab4d9a846c19ae6e4991e7, 7bffd54a4b224a75b78337fd20bc4c14
I20260619 06:58:29.238551 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.238291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:53132)
0619 06:58:29.238412 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.238488 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 06:58:29.238590 19121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:58:29.238909 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6f91ecc0c3184b5282e4146af78161b4"
I20260619 06:58:29.239158 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.238947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:53148)
0619 06:58:29.239047 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.239096 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 06:58:29.239312 18394 leader_election.cc:334] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7bffd54a4b224a75b78337fd20bc4c14 (127.11.106.126:34885): Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
I20260619 06:58:29.239727 19121 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 06:58:29.240689 18605 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46bf7f0b36ab4d9a846c19ae6e4991e7 in term 2.
I20260619 06:58:29.240998 18393 leader_election.cc:302] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46bf7f0b36ab4d9a846c19ae6e4991e7, 6f91ecc0c3184b5282e4146af78161b4; no voters: 7bffd54a4b224a75b78337fd20bc4c14
I20260619 06:58:29.241161 19122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:58:29.241366 19122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [term 2 LEADER]: Becoming Leader. State: Replica: 46bf7f0b36ab4d9a846c19ae6e4991e7, State: Running, Role: LEADER
I20260619 06:58:29.241618 19122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } }
I20260619 06:58:29.242899 19131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46bf7f0b36ab4d9a846c19ae6e4991e7. Latest consensus state: current_term: 2 leader_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:29.242997 19131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:29.243314 19133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:29.245708 19133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:29.246181 19133 catalog_manager.cc:1259] Loaded cluster ID: de7075af517749e9b4831371afb48188
I20260619 06:58:29.246248 19133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:29.246949 19133 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:29.247335 19133 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Loaded TSK: 0
I20260619 06:58:29.248014 19133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:58:29.277486 18408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:29.280694 18408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:29.284072 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.283757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:53178)
0619 06:58:29.283913 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.283983 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 06:58:29.284140 18605 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [term 2 FOLLOWER]: Refusing update from remote peer 46bf7f0b36ab4d9a846c19ae6e4991e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 06:58:29.284706 19131 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 06:58:29.285061 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.284803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.0.0.1:53186)
0619 06:58:29.284939 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.284996 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 06:58:29.285192 18394 consensus_peers.cc:487] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 -> Peer 7bffd54a4b224a75b78337fd20bc4c14 (127.11.106.126:34885): Couldn't send request to peer 7bffd54a4b224a75b78337fd20bc4c14. Status: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:58:29.286334 19121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46bf7f0b36ab4d9a846c19ae6e4991e7. Latest consensus state: current_term: 2 leader_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:29.286466 19121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:58:29.287758 19122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:29.287894 19122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:29.288456 19121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7bffd54a4b224a75b78337fd20bc4c14" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 34885 } } peers { permanent_uuid: "46bf7f0b36ab4d9a846c19ae6e4991e7" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 39157 } attrs { promote: false } } peers { permanent_uuid: "6f91ecc0c3184b5282e4146af78161b4" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 43591 } attrs { promote: false } } }
I20260619 06:58:29.288620 19121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f91ecc0c3184b5282e4146af78161b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:58:29.298691 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.294616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:39931 (local address 127.0.0.1:55524)
0619 06:58:29.294888 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.294903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:29.294990 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:29.295521 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:29.295524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:29.295542 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:29.295636 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.295642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.296713 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.296747 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.297515 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.297523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.298194 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.298198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.298336 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.298361 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:58:29.298433 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 06:58:29.298506 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":7,"thread_start_us":102,"threads_started":1}
I20260619 06:58:29.299242 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.294988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.11.106.67:39931)
0619 06:58:29.295239 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:29.295244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:29.295257 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:58:29.295308 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:29.295311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:29.295353 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:29.295742 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:29.295851 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.296603 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.297643 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.298071 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.298184 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.298261 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:29.299061 (+ 800us) server_negotiation.cc:299] Negotiation successful
0619 06:58:29.299138 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260619 06:58:29.299789 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.294122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:42889 (local address 127.0.0.1:60176)
0619 06:58:29.294444 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.294459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:29.294551 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:29.295847 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:29.295852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:29.295871 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:29.295981 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.295987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.297069 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.297074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.298310 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.298320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.299342 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.299348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.299458 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.299483 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:58:29.299540 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 06:58:29.299616 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":32,"spinlock_wait_cycles":4352}
I20260619 06:58:29.299796 18986 tablet_service.cc:1467] Processing CreateTablet for tablet 30e3aaf2a4754c02bc44ab06d98d889b (DEFAULT_TABLE table=test-table [id=284a4ec1bde04820841c1acb3fc8659b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:29.300119 18986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e3aaf2a4754c02bc44ab06d98d889b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:29.300125 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.294325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:36035 (local address 127.0.0.1:47360)
0619 06:58:29.294465 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.294479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:29.294551 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:29.295226 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:29.295230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:29.295252 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:29.295361 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.295368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.296602 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.296606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.298377 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.298388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.299722 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.299727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.299823 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.299853 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:58:29.299911 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 06:58:29.299996 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":419,"spinlock_wait_cycles":2816}
I20260619 06:58:29.300417 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.294258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.11.106.66:42889)
0619 06:58:29.295477 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:29.295482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:29.295497 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:29.295550 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:29.295553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:29.295601 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:29.296076 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:29.296166 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.296929 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.298487 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.299200 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.300063 (+ 863us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.300130 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:29.300171 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 06:58:29.300251 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":84,"threads_started":1}
I20260619 06:58:29.300923 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.294698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.11.106.65:36035)
0619 06:58:29.294960 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:29.294965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:29.294979 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:58:29.295032 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:29.295038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:29.295087 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:29.295206 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:29.295628 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.296453 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.298513 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.299591 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.300404 (+ 813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.300485 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:29.300749 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 06:58:29.300819 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260619 06:58:29.300931 18850 tablet_service.cc:1467] Processing CreateTablet for tablet 30e3aaf2a4754c02bc44ab06d98d889b (DEFAULT_TABLE table=test-table [id=284a4ec1bde04820841c1acb3fc8659b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:29.301167 18850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e3aaf2a4754c02bc44ab06d98d889b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:29.301442 18713 tablet_service.cc:1467] Processing CreateTablet for tablet 30e3aaf2a4754c02bc44ab06d98d889b (DEFAULT_TABLE table=test-table [id=284a4ec1bde04820841c1acb3fc8659b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:29.301733 18713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e3aaf2a4754c02bc44ab06d98d889b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:29.302953 19144 tablet_bootstrap.cc:492] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: Bootstrap starting.
I20260619 06:58:29.303678 19145 tablet_bootstrap.cc:492] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20: Bootstrap starting.
I20260619 06:58:29.303995 19144 tablet_bootstrap.cc:654] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:29.304519 19145 tablet_bootstrap.cc:654] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:29.304522 19146 tablet_bootstrap.cc:492] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a: Bootstrap starting.
I20260619 06:58:29.305516 19146 tablet_bootstrap.cc:654] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:29.305595 19144 tablet_bootstrap.cc:492] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: No bootstrap required, opened a new log
I20260619 06:58:29.305709 19144 ts_tablet_manager.cc:1397] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:29.305894 19145 tablet_bootstrap.cc:492] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20: No bootstrap required, opened a new log
I20260619 06:58:29.305989 19145 ts_tablet_manager.cc:1397] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:58:29.306201 19144 raft_consensus.cc:348] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } }
I20260619 06:58:29.306349 19144 raft_consensus.cc:374] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:29.306398 19144 raft_consensus.cc:729] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0af17d04af45ca901f90a9ade63781, State: Initialized, Role: FOLLOWER
I20260619 06:58:29.306483 19145 raft_consensus.cc:348] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } }
I20260619 06:58:29.306550 19144 consensus_queue.cc:260] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } }
I20260619 06:58:29.306618 19145 raft_consensus.cc:374] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:29.306654 19145 raft_consensus.cc:729] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5411bf416f874e9b9acfbaa08c340e20, State: Initialized, Role: FOLLOWER
I20260619 06:58:29.306792 19145 consensus_queue.cc:260] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } }
I20260619 06:58:29.306945 19144 ts_tablet_manager.cc:1428] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 06:58:29.307142 19145 ts_tablet_manager.cc:1428] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 06:58:29.308123 19146 tablet_bootstrap.cc:492] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a: No bootstrap required, opened a new log
I20260619 06:58:29.308228 19146 ts_tablet_manager.cc:1397] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 06:58:29.308699 19146 raft_consensus.cc:348] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } }
I20260619 06:58:29.308811 19146 raft_consensus.cc:374] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:29.308851 19146 raft_consensus.cc:729] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea6eac2761c4cc5a736687209d01f3a, State: Initialized, Role: FOLLOWER
I20260619 06:58:29.308991 19146 consensus_queue.cc:260] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } }
I20260619 06:58:29.309862 19146 ts_tablet_manager.cc:1428] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:29.310303 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.309581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.65:44591)
0619 06:58:29.310114 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.310178 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":87,"threads_started":1}
I20260619 06:58:29.311121 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.310731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.65:44301)
0619 06:58:29.311007 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.311057 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260619 06:58:29.311131 18781 heartbeater.cc:503] Master 127.11.106.125:39157 was elected leader, sending a full tablet report...
I20260619 06:58:29.312698 19054 heartbeater.cc:503] Master 127.11.106.125:39157 was elected leader, sending a full tablet report...
I20260619 06:58:29.313594 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.313153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.67:44953)
0619 06:58:29.313425 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.313490 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":117,"threads_started":1}
I20260619 06:58:29.313843 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.308824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.66:48715)
0619 06:58:29.313584 (+ 4760us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.313642 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4679,"thread_start_us":106,"threads_started":1}
I20260619 06:58:29.314554 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.314330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.66:58391)
0619 06:58:29.314445 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.314495 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 06:58:29.541276 19151 raft_consensus.cc:482] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:29.541428 19151 raft_consensus.cc:504] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } }
I20260619 06:58:29.542562 19151 leader_election.cc:288] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ea6eac2761c4cc5a736687209d01f3a (127.11.106.65:36035), 7c0af17d04af45ca901f90a9ade63781 (127.11.106.67:39931)
I20260619 06:58:29.547469 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.543392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:54803 (local address 127.11.106.67:39931)
0619 06:58:29.543560 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:29.543563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:29.543670 (+ 107us) server_negotiation.cc:407] Connection header received
0619 06:58:29.543778 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:29.543781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:29.543835 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:29.543903 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:29.544183 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.544974 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.546408 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.546857 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.547009 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.547068 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:29.547241 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 06:58:29.547333 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:58:29.547483 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.543265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:39931 (local address 127.11.106.66:54803)
0619 06:58:29.543577 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.543588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:29.543667 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:29.543938 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:29.543940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:29.543957 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:29.544064 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.544070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.545126 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.545130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.546285 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.546296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.547009 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.547014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.547107 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.547130 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:58:29.547225 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:58:29.547308 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":62,"thread_start_us":131,"threads_started":1}
I20260619 06:58:29.547636 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.542925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:36035 (local address 127.11.106.66:44581)
0619 06:58:29.543092 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.543131 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:29.543279 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:29.543755 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:29.543760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:29.543780 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:29.543924 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.543930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.545107 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.545114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.546283 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:29.546294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.547090 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.547094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:29.547249 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.547275 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 06:58:29.547352 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:58:29.547431 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":99}
I20260619 06:58:29.548019 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.543067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:44581 (local address 127.11.106.65:36035)
0619 06:58:29.543225 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:29.543232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:29.543249 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:58:29.543412 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:29.543416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:29.543611 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:29.543722 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:29.544049 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.544957 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.546418 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.546963 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.547107 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.547163 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:29.547525 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 06:58:29.547880 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:58:29.548370 19006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e3aaf2a4754c02bc44ab06d98d889b" candidate_uuid: "5411bf416f874e9b9acfbaa08c340e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0af17d04af45ca901f90a9ade63781" is_pre_election: true
I20260619 06:58:29.548401 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e3aaf2a4754c02bc44ab06d98d889b" candidate_uuid: "5411bf416f874e9b9acfbaa08c340e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6eac2761c4cc5a736687209d01f3a" is_pre_election: true
I20260619 06:58:29.548609 18733 raft_consensus.cc:2454] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5411bf416f874e9b9acfbaa08c340e20 in term 0.
I20260619 06:58:29.548698 19006 raft_consensus.cc:2454] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5411bf416f874e9b9acfbaa08c340e20 in term 0.
I20260619 06:58:29.549074 18803 leader_election.cc:302] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6eac2761c4cc5a736687209d01f3a, 5411bf416f874e9b9acfbaa08c340e20; no voters:
I20260619 06:58:29.549358 19151 raft_consensus.cc:2781] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:29.549422 19151 raft_consensus.cc:482] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:29.549461 19151 raft_consensus.cc:3037] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:29.550555 19151 raft_consensus.cc:504] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } }
I20260619 06:58:29.550917 19151 leader_election.cc:288] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [CANDIDATE]: Term 1 election: Requested vote from peers 0ea6eac2761c4cc5a736687209d01f3a (127.11.106.65:36035), 7c0af17d04af45ca901f90a9ade63781 (127.11.106.67:39931)
I20260619 06:58:29.551376 19006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e3aaf2a4754c02bc44ab06d98d889b" candidate_uuid: "5411bf416f874e9b9acfbaa08c340e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0af17d04af45ca901f90a9ade63781"
I20260619 06:58:29.551385 18733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e3aaf2a4754c02bc44ab06d98d889b" candidate_uuid: "5411bf416f874e9b9acfbaa08c340e20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6eac2761c4cc5a736687209d01f3a"
I20260619 06:58:29.551488 19006 raft_consensus.cc:3037] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:29.551518 18733 raft_consensus.cc:3037] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:29.552421 18733 raft_consensus.cc:2454] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5411bf416f874e9b9acfbaa08c340e20 in term 1.
I20260619 06:58:29.552516 19006 raft_consensus.cc:2454] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5411bf416f874e9b9acfbaa08c340e20 in term 1.
I20260619 06:58:29.552801 18803 leader_election.cc:302] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea6eac2761c4cc5a736687209d01f3a, 5411bf416f874e9b9acfbaa08c340e20; no voters:
I20260619 06:58:29.553117 19151 raft_consensus.cc:2781] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:29.553356 19151 raft_consensus.cc:686] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 1 LEADER]: Becoming Leader. State: Replica: 5411bf416f874e9b9acfbaa08c340e20, State: Running, Role: LEADER
I20260619 06:58:29.553597 18918 heartbeater.cc:503] Master 127.11.106.125:39157 was elected leader, sending a full tablet report...
I20260619 06:58:29.553673 19151 consensus_queue.cc:237] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } }
I20260619 06:58:29.554173 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.553903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.66:45373)
0619 06:58:29.554033 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.554105 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:58:29.556404 18409 catalog_manager.cc:5495] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5411bf416f874e9b9acfbaa08c340e20 (127.11.106.66). New cstate: current_term: 1 leader_uuid: "5411bf416f874e9b9acfbaa08c340e20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5411bf416f874e9b9acfbaa08c340e20" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 42889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:29.598088 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.594309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.11.106.66:42889)
0619 06:58:29.594457 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:29.594464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:29.594494 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:58:29.594584 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:29.594587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:29.594641 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:29.594762 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:29.595027 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.595622 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.596592 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:29.596824 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:29.596937 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:29.597218 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:29.597671 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:29.597817 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:29.597879 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 06:58:29.597939 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:58:29.601249 18733 raft_consensus.cc:1261] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a [term 1 FOLLOWER]: Refusing update from remote peer 5411bf416f874e9b9acfbaa08c340e20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:29.601524 19006 raft_consensus.cc:1261] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [term 1 FOLLOWER]: Refusing update from remote peer 5411bf416f874e9b9acfbaa08c340e20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:29.601914 19151 consensus_queue.cc:1035] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea6eac2761c4cc5a736687209d01f3a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:29.602155 19158 consensus_queue.cc:1035] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0af17d04af45ca901f90a9ade63781" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:29.604590 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.604297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.67:39881)
0619 06:58:29.604398 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.604470 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 06:58:29.606202 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.605938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.65:35573)
0619 06:58:29.606061 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.606122 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 06:58:29.608165 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.607875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.66:44965)
0619 06:58:29.608027 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.608087 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 06:58:29.608727 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.608431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.67:35549)
0619 06:58:29.608598 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.608653 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:58:29.609304 19163 mvcc.cc:205] Tried to move back new op lower bound from 7298467060120809472 to 7298467059934822400. Current Snapshot: MvccSnapshot[applied={T|T < 7298467060120809472}]
I20260619 06:58:29.611590 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.610889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:34885 (local address 127.11.106.65:35301)
0619 06:58:29.611443 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.611514 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466}
I20260619 06:58:29.616843 18409 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:29.617065 18409 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:29.621634 18409 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46bf7f0b36ab4d9a846c19ae6e4991e7: Sending DeleteTablet for 3 replicas of tablet 30e3aaf2a4754c02bc44ab06d98d889b
I20260619 06:58:29.622531 18713 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e3aaf2a4754c02bc44ab06d98d889b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:29 UTC) from {username='oryx', principal='oryx/127.11.106.125@KRBTEST.COM'} at 127.0.0.1:47360
I20260619 06:58:29.622956 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 18553
I20260619 06:58:29.623052 18986 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e3aaf2a4754c02bc44ab06d98d889b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:29 UTC) from {username='oryx', principal='oryx/127.11.106.125@KRBTEST.COM'} at 127.0.0.1:55524
I20260619 06:58:29.623149 19171 tablet_replica.cc:330] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a: stopping tablet replica
I20260619 06:58:29.623302 19171 raft_consensus.cc:2229] T 30e3aaf2a4754c02bc44ab06d98d889b P 0ea6eac2761c4cc5a736687209d01f3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:29.623476 18850 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e3aaf2a4754c02bc44ab06d98d889b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:29 UTC) from {username='oryx', principal='oryx/127.11.106.125@KRBTEST.COM'} at 127.0.0.1:60176
I20260619 06:58:29.623848 19172 tablet_replica.cc:330] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: stopping tablet replica
I20260619 06:58:29.623975 19172 raft_consensus.cc:2229] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:29.624099 19172 raft_consensus.cc:2258] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:29.624905 19172 ts_tablet_manager.cc:1905] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:29.627626 19172 ts_tablet_manager.cc:1918] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:29.627696 19172 log.cc:1196] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/30e3aaf2a4754c02bc44ab06d98d889b
I20260619 06:58:29.627990 19172 ts_tablet_manager.cc:1939] T 30e3aaf2a4754c02bc44ab06d98d889b P 7c0af17d04af45ca901f90a9ade63781: Deleting consensus metadata
I20260619 06:58:29.628883 18393 catalog_manager.cc:4841] TS 7c0af17d04af45ca901f90a9ade63781 (127.11.106.67:39931): tablet 30e3aaf2a4754c02bc44ab06d98d889b (table test-table [id=284a4ec1bde04820841c1acb3fc8659b]) successfully deleted
I20260619 06:58:29.631289 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 18787
I20260619 06:58:29.631348 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.631041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:36035 (local address 127.0.0.1:47372)
0619 06:58:29.631203 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.631257 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:36035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 06:58:29.631542 19173 tablet_replica.cc:330] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20: stopping tablet replica
I20260619 06:58:29.631739 19173 raft_consensus.cc:2229] T 30e3aaf2a4754c02bc44ab06d98d889b P 5411bf416f874e9b9acfbaa08c340e20 [term 1 LEADER]: Raft consensus shutting down.
W20260619 06:58:29.632330 18393 catalog_manager.cc:4584] TS 0ea6eac2761c4cc5a736687209d01f3a (127.11.106.65:36035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30e3aaf2a4754c02bc44ab06d98d889b: Network error: Client connection negotiation failed: client connection to 127.11.106.65:36035: connect: Connection refused (error 111)
I20260619 06:58:29.637986 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 18923
I20260619 06:58:29.638594 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:29.638095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:42889 (local address 127.0.0.1:60200)
0619 06:58:29.638419 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:29.638491 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.66:42889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 06:58:29.645387 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 18273
I20260619 06:58:29.652105 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 18373
2026-06-19T06:58:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5619 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:29 dist-test-slave-10d0 krb5kdc[19176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:29 dist-test-slave-10d0 krb5kdc[19176](info): set up 2 sockets
Jun 19 06:58:29 dist-test-slave-10d0 krb5kdc[19176](info): commencing operation
krb5kdc: starting...
W20260619 06:58:30.734167 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:30 dist-test-slave-10d0 krb5kdc[19176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.126.keytab.
Entry for principal oryx/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.126.keytab.
I20260619 06:58:30.865506 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36967
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.126.keytab
--principal=oryx/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:30.979389 19189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:30.979697 19189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:30.979763 19189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:30.983773 19189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:30.983867 19189 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:58:30.983902 19189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:30.983949 19189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:30.983990 19189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:30.984027 19189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:30.989372 19189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:36967
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.126.keytab
--principal=oryx/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:30.990486 19189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:30.991552 19189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:31.000275 19194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:31.000291 19197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:31.000275 19195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:31.000345 19189 server_base.cc:1034] running on GCE node
I20260619 06:58:31.001184 19189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:31.002360 19189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:31.003595 19189 hybrid_clock.cc:648] HybridClock initialized: now 1781852311003581 us; error 39 us; skew 500 ppm
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:31.006772 19189 init.cc:377] Logged in from keytab as oryx/127.11.106.126@KRBTEST.COM (short username oryx)
I20260619 06:58:31.008129 19189 webserver.cc:460] Webserver started at http://127.11.106.126:43677/ using document root <none> and password file <none>
I20260619 06:58:31.008809 19189 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:31.008900 19189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:31.009141 19189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:31.011276 19189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "2a94c8f6cc854c4094bcf884cc875e29"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "af0f43e29a3e13c36b04a911591a58a7"
server_key_iv: "b40f88ae7f8d641c0881de6811f62d79"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.011921 19189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "2a94c8f6cc854c4094bcf884cc875e29"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "af0f43e29a3e13c36b04a911591a58a7"
server_key_iv: "b40f88ae7f8d641c0881de6811f62d79"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.016176 19189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:58:31.018896 19204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:31.020304 19189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:31.020478 19189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "2a94c8f6cc854c4094bcf884cc875e29"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "af0f43e29a3e13c36b04a911591a58a7"
server_key_iv: "b40f88ae7f8d641c0881de6811f62d79"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.020678 19189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:31.042963 19189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:31.046512 19189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:31.046741 19189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:31.055235 19189 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:36967
I20260619 06:58:31.055240 19266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:36967 every 8 connection(s)
I20260619 06:58:31.056249 19189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:31.059298 19267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:31.062661 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 19189
I20260619 06:58:31.062848 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:31.063124 11689 external_mini_cluster.cc:1428] Setting key 852569c8b01439e9412e833b7330728d
I20260619 06:58:31.066104 19267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29: Bootstrap starting.
I20260619 06:58:31.069093 19267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:31.069984 19267 log.cc:824] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:31.072371 19267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29: No bootstrap required, opened a new log
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:31.077527 19267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94c8f6cc854c4094bcf884cc875e29" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36967 } }
I20260619 06:58:31.077855 19267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:31.077929 19267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a94c8f6cc854c4094bcf884cc875e29, State: Initialized, Role: FOLLOWER
I20260619 06:58:31.078533 19267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94c8f6cc854c4094bcf884cc875e29" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36967 } }
I20260619 06:58:31.078737 19267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:31.078814 19267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:31.078918 19267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:31.079629 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.064655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.11.106.126:36967)
0619 06:58:31.065116 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.065138 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.065185 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:58:31.066068 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.066096 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.066560 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.066960 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:31.067664 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.069373 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.070336 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.071042 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.071214 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.073830 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:31.073837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:31.073855 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:31.073890 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:31.076085 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.076578 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.076587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.076595 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.076766 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.077055 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.077058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.077060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.077608 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:31.077838 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.078268 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.078683 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":106,"threads_started":1}
I20260619 06:58:31.080036 19267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94c8f6cc854c4094bcf884cc875e29" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36967 } }
I20260619 06:58:31.080415 19267 leader_election.cc:302] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a94c8f6cc854c4094bcf884cc875e29; no voters:
I20260619 06:58:31.080860 19267 leader_election.cc:288] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:31.080943 19272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:31.081164 19272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [term 1 LEADER]: Becoming Leader. State: Replica: 2a94c8f6cc854c4094bcf884cc875e29, State: Running, Role: LEADER
I20260619 06:58:31.081489 19272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94c8f6cc854c4094bcf884cc875e29" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36967 } }
I20260619 06:58:31.082307 19267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:31.083460 19274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a94c8f6cc854c4094bcf884cc875e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94c8f6cc854c4094bcf884cc875e29" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36967 } } }
I20260619 06:58:31.083498 19273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a94c8f6cc854c4094bcf884cc875e29. Latest consensus state: current_term: 1 leader_uuid: "2a94c8f6cc854c4094bcf884cc875e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a94c8f6cc854c4094bcf884cc875e29" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36967 } } }
I20260619 06:58:31.083737 19273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:31.083766 19274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:31.084178 19281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:31.087677 19281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:31.093948 19281 catalog_manager.cc:1347] Generated new cluster ID: d9d51dd7a196412d853c280b0add8600
I20260619 06:58:31.094067 19281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:31.111271 19281 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:31.112665 19281 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:31.127010 19281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29: Generated new TSK 0
I20260619 06:58:31.127970 19281 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab.
Entry for principal oryx/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab.
I20260619 06:58:31.156708 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36967
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab
--principal=oryx/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:31.270331 19293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:31.270663 19293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:31.270747 19293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:31.274930 19293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:31.275022 19293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:31.275096 19293 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:58:31.275178 19293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:31.280668 19293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab
--principal=oryx/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.106.126:36967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:31.281864 19293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:31.283079 19293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:31.293062 19301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:31.293107 19298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:31.293159 19293 server_base.cc:1034] running on GCE node
W20260619 06:58:31.293077 19299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:31.293889 19293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:31.294569 19293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:31.295773 19293 hybrid_clock.cc:648] HybridClock initialized: now 1781852311295737 us; error 49 us; skew 500 ppm
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:31.298580 19293 init.cc:377] Logged in from keytab as oryx/127.11.106.65@KRBTEST.COM (short username oryx)
I20260619 06:58:31.299588 19293 webserver.cc:460] Webserver started at http://127.11.106.65:45265/ using document root <none> and password file <none>
I20260619 06:58:31.300289 19293 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:31.300352 19293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:31.300652 19293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:31.302706 19293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "5a256df638b34b75a893c6e1056fc3c6"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "0569d9c0d290915b3fc779949c3c8ea4"
server_key_iv: "e25e1863282d0aea39b70db19d4f0688"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.303278 19293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "5a256df638b34b75a893c6e1056fc3c6"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "0569d9c0d290915b3fc779949c3c8ea4"
server_key_iv: "e25e1863282d0aea39b70db19d4f0688"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.307138 19293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:31.309638 19308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:31.310837 19293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:31.310935 19293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "5a256df638b34b75a893c6e1056fc3c6"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "0569d9c0d290915b3fc779949c3c8ea4"
server_key_iv: "e25e1863282d0aea39b70db19d4f0688"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.311071 19293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:31.332120 19293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:31.335244 19293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:31.335465 19293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:31.337080 19293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:31.337158 19293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:31.337232 19293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:31.337283 19293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:31.348646 19293 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:41247
I20260619 06:58:31.348718 19423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:41247 every 8 connection(s)
I20260619 06:58:31.349725 19293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:31.351320 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.337877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:38783 (local address 127.11.106.126:36967)
0619 06:58:31.338031 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.338036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.339099 (+ 1063us) server_negotiation.cc:407] Connection header received
0619 06:58:31.340513 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.340518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.340605 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.340704 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:31.342054 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.342660 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.343915 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.344257 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.344378 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.347391 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:31.347395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:31.347398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:31.347425 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:31.349094 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.349662 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.349665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.349666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.349779 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.350158 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.350164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.350169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.350395 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:31.350493 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.351086 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.351199 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:58:31.352088 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.338387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36967 (local address 127.11.106.65:38783)
0619 06:58:31.338927 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:31.338988 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:31.340248 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:31.340924 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:31.340933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:31.341274 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:31.341892 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.341928 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.342789 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.342793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.343786 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.343795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.344439 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.344443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.344670 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.345299 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:31.345323 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:31.347199 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:31.349206 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.349212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.349226 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.349559 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.349913 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.349916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.349918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.350059 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.350547 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:31.350553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:31.350841 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 06:58:31.351000 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 06:58:31.351269 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":142,"threads_started":1}
I20260619 06:58:31.353411 19424 heartbeater.cc:347] Connected to a master server at 127.11.106.126:36967
I20260619 06:58:31.353554 19424 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:31.353935 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 19293
I20260619 06:58:31.354075 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:31.354120 19424 heartbeater.cc:511] Master 127.11.106.126:36967 requested a full tablet report, sending...
I20260619 06:58:31.354423 11689 external_mini_cluster.cc:1428] Setting key 2f43f3eaf8babb7115ed53beb616a48e
I20260619 06:58:31.355842 19220 ts_manager.cc:194] Registered new tserver with Master: 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247)
I20260619 06:58:31.357489 19220 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:38783
WARNING: no policy specified for oryx/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 06:58:31.366273 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.355742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36967 (local address 127.11.106.65:33847)
0619 06:58:31.357954 (+ 2212us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:31.357970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:31.358081 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:31.358588 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:31.358593 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:31.358866 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:31.359043 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.359056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.360201 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.360207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.361316 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.361332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.361808 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.361815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.361951 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.362503 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:31.362523 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:31.363018 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:31.364814 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.364820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.364825 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.365101 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.365378 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.365381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.365382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.365469 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.365824 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:31.365829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:31.365923 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 06:58:31.366006 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 06:58:31.366119 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2122,"thread_start_us":104,"threads_started":1}
I20260619 06:58:31.366309 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.355773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:33847 (local address 127.11.106.126:36967)
0619 06:58:31.355899 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.355903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.358069 (+ 2166us) server_negotiation.cc:407] Connection header received
0619 06:58:31.358283 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.358288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.358361 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.358452 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:31.359189 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.360031 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.361453 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.361666 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.361765 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.363166 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:31.363169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:31.363171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:31.363205 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:31.364708 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.365210 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.365213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.365214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.365288 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.365547 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.365549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.365550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.365711 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:31.365787 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.366087 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.366197 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab.
Entry for principal oryx/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab.
I20260619 06:58:31.384521 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36967
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab
--principal=oryx/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:31.495378 19433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:31.495644 19433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:31.495689 19433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:31.499696 19433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:31.499759 19433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:31.499842 19433 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:58:31.499905 19433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:31.505326 19433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab
--principal=oryx/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.106.126:36967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:31.506354 19433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:31.507346 19433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:31.517201 19439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:31.517256 19433 server_base.cc:1034] running on GCE node
W20260619 06:58:31.517187 19438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:31.517187 19441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:31.517855 19433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:31.518514 19433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:31.519690 19433 hybrid_clock.cc:648] HybridClock initialized: now 1781852311519672 us; error 29 us; skew 500 ppm
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:31.522466 19433 init.cc:377] Logged in from keytab as oryx/127.11.106.66@KRBTEST.COM (short username oryx)
I20260619 06:58:31.523268 19433 webserver.cc:460] Webserver started at http://127.11.106.66:42945/ using document root <none> and password file <none>
I20260619 06:58:31.523801 19433 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:31.523882 19433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:31.524106 19433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:31.526293 19433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "7735208f2f6a48898fb3da5767c9a7ce"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "ede94404d9422fa5be7d55be95068c82"
server_key_iv: "30fde2c7deae536f0637c4571cb93b1e"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.526829 19433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "7735208f2f6a48898fb3da5767c9a7ce"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "ede94404d9422fa5be7d55be95068c82"
server_key_iv: "30fde2c7deae536f0637c4571cb93b1e"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.530825 19433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:31.533422 19448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:31.534799 19433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:31.534950 19433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "7735208f2f6a48898fb3da5767c9a7ce"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "ede94404d9422fa5be7d55be95068c82"
server_key_iv: "30fde2c7deae536f0637c4571cb93b1e"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.535079 19433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:31.549530 19433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:31.553295 19433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:31.553507 19433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:31.555116 19433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:31.555198 19433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:31.555259 19433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:31.555289 19433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:31.566509 19433 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:34551
I20260619 06:58:31.566623 19563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:34551 every 8 connection(s)
I20260619 06:58:31.567545 19433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:31.569859 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.556194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:53771 (local address 127.11.106.126:36967)
0619 06:58:31.556356 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.556361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.557361 (+ 1000us) server_negotiation.cc:407] Connection header received
0619 06:58:31.558754 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.558758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.558831 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.558939 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:31.560593 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.561510 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.562938 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.563194 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.563315 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.566003 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:31.566007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:31.566010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:31.566046 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:31.567673 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.568300 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.568304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.568306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.568398 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.568860 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.568864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.568865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.569058 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:31.569148 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.569626 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.569740 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:58:31.570719 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.556643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36967 (local address 127.11.106.66:53771)
0619 06:58:31.557161 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:31.557234 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:31.558451 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:31.559171 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:31.559182 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:31.559608 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:31.560372 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.560391 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.561678 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.561682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.562805 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.562814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.563355 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.563359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.563546 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.564187 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:31.564204 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:31.565821 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:31.567797 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.567803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.567819 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.568161 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.568512 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.568514 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.568516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.568731 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.569167 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:31.569172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:31.569405 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 06:58:31.569558 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 06:58:31.569857 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260619 06:58:31.571262 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 19433
I20260619 06:58:31.571395 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:31.571740 11689 external_mini_cluster.cc:1428] Setting key c7c36e2ef368058f94577f94bf2ca6a8
I20260619 06:58:31.572117 19564 heartbeater.cc:347] Connected to a master server at 127.11.106.126:36967
I20260619 06:58:31.572270 19564 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:31.573348 19564 heartbeater.cc:511] Master 127.11.106.126:36967 requested a full tablet report, sending...
I20260619 06:58:31.574375 19221 ts_manager.cc:194] Registered new tserver with Master: 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551)
I20260619 06:58:31.575079 19221 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:53771
WARNING: no policy specified for oryx/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 06:58:31.582880 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.574511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:54821 (local address 127.11.106.126:36967)
0619 06:58:31.574620 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.574624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.574936 (+ 312us) server_negotiation.cc:407] Connection header received
0619 06:58:31.575061 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.575064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.575115 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.575211 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:31.575766 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.576498 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.577597 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.578007 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.578126 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.579351 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:31.579356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:31.579362 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:31.579395 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:31.581076 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.581482 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.581486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.581488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.581586 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.581931 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.581937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.581944 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.582152 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:31.582278 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.582599 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.582712 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:58:31.582891 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.574437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36967 (local address 127.11.106.66:54821)
0619 06:58:31.574837 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:31.574854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:31.574962 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:31.575231 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:31.575235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:31.575488 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:31.575638 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.575648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.576681 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.576685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.577458 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.577468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.578129 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.578131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.578209 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.578821 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:31.578839 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:31.579198 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:31.581182 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.581184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.581185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.581377 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.581743 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.581745 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.581746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.581820 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.582323 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:31.582327 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:31.582404 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 06:58:31.582541 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 06:58:31.582692 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab.
Entry for principal oryx/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab.
I20260619 06:58:31.599992 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36967
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab
--principal=oryx/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:31.713795 19573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:31.714100 19573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:31.714175 19573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:31.718452 19573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:31.718541 19573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:31.718613 19573 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:58:31.718706 19573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:31.724222 19573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab
--principal=oryx/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.106.126:36967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:31.725479 19573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:31.726549 19573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:31.736333 19578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:31.736330 19581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:31.736420 19573 server_base.cc:1034] running on GCE node
W20260619 06:58:31.736369 19579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:31.736961 19573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:31.737664 19573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:31.738859 19573 hybrid_clock.cc:648] HybridClock initialized: now 1781852311738805 us; error 67 us; skew 500 ppm
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:31.741737 19573 init.cc:377] Logged in from keytab as oryx/127.11.106.67@KRBTEST.COM (short username oryx)
I20260619 06:58:31.742569 19573 webserver.cc:460] Webserver started at http://127.11.106.67:46037/ using document root <none> and password file <none>
I20260619 06:58:31.743088 19573 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:31.743144 19573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:31.743319 19573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:31.745385 19573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "ff64b894c21546f0b654955dab62df7b"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "0c5eb9d407b2bfb4f20be8f3e181133c"
server_key_iv: "c6e8b976a7dcbfd7b9e58629b6db3ca4"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.745867 19573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "ff64b894c21546f0b654955dab62df7b"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "0c5eb9d407b2bfb4f20be8f3e181133c"
server_key_iv: "c6e8b976a7dcbfd7b9e58629b6db3ca4"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.749614 19573 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:58:31.752108 19588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:31.753587 19573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:31.753716 19573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "ff64b894c21546f0b654955dab62df7b"
format_stamp: "Formatted at 2026-06-19 06:58:31 on dist-test-slave-10d0"
server_key: "0c5eb9d407b2bfb4f20be8f3e181133c"
server_key_iv: "c6e8b976a7dcbfd7b9e58629b6db3ca4"
server_key_version: "encryptionkey@0"
I20260619 06:58:31.753829 19573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:31.765252 19573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:31.768743 19573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:31.768955 19573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:31.770458 19573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:31.770509 19573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:31.770550 19573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:31.770608 19573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:31.781512 19573 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:37521
I20260619 06:58:31.781584 19703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:37521 every 8 connection(s)
I20260619 06:58:31.782596 19573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:31.785959 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.771517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:48229 (local address 127.11.106.126:36967)
0619 06:58:31.771681 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.771696 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.772602 (+ 906us) server_negotiation.cc:407] Connection header received
0619 06:58:31.773675 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.773681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.773760 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.773876 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:31.775597 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.776517 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.778515 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.778741 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.778828 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.782055 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:31.782058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:31.782060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:31.782086 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:31.783865 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.784456 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.784459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.784460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.784558 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.784915 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.784919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.784920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.785127 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:31.785221 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.785721 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.785834 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:58:31.786769 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 19573
I20260619 06:58:31.786773 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.771901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36967 (local address 127.11.106.67:48229)
0619 06:58:31.772394 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:31.772446 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:31.773426 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:31.774147 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:31.774156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:31.774673 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:31.775400 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.775423 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.776739 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.776744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.778377 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.778389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.778918 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.778923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.779140 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.779906 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:31.779935 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:31.781840 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:31.783993 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.784000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.784020 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.784363 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.784639 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.784641 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.784643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.784785 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.785241 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:31.785246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:31.785474 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 06:58:31.785631 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 06:58:31.785889 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":147,"threads_started":1}
I20260619 06:58:31.786893 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:31.787206 11689 external_mini_cluster.cc:1428] Setting key 267493fe2d98959ed821c2d9cbab3916
I20260619 06:58:31.788228 19704 heartbeater.cc:347] Connected to a master server at 127.11.106.126:36967
I20260619 06:58:31.788367 19704 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:31.789016 19704 heartbeater.cc:511] Master 127.11.106.126:36967 requested a full tablet report, sending...
I20260619 06:58:31.790097 19220 ts_manager.cc:194] Registered new tserver with Master: ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521)
I20260619 06:58:31.790844 19220 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:48229
I20260619 06:58:31.797564 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.790961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:43235 (local address 127.11.106.126:36967)
0619 06:58:31.791100 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.791104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.791234 (+ 130us) server_negotiation.cc:407] Connection header received
0619 06:58:31.791380 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.791383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.791430 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.791512 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:31.791997 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.792492 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.793522 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.793759 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.793842 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.794819 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:31.794822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:31.794824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:31.794844 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:31.796045 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.796418 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.796421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.796422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.796491 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.796803 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.796806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.796807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.796991 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:31.797080 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.797352 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.797449 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:58:31.797657 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.790844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36967 (local address 127.11.106.67:43235)
0619 06:58:31.791140 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:31.791156 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:31.791277 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:31.791527 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:31.791529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:31.791745 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:31.791859 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.791866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.792655 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.792659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.793407 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.793415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.793857 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.793860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.793929 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.794368 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:31.794380 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:31.794698 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:31.796151 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.796154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.796155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.796336 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.796613 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:31.796616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:31.796618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:31.796700 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:31.797110 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:31.797113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:31.797191 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 06:58:31.797297 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:58:31.797464 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":119,"threads_started":1}
I20260619 06:58:31.801170 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:58:31.825804 19270 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:31.818859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.11.106.126:36967)
0619 06:58:31.819023 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.819031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.819052 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:58:31.819128 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.819130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.819177 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.819277 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:31.820147 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.820758 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.821661 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.821890 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.822018 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.825614 (+ 3596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42852: BlockingRecv error: recv got EOF from 127.0.0.1:42852 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 06:58:31 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:31.834978 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.827201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.11.106.126:36967)
0619 06:58:31.827330 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.827337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.827523 (+ 186us) server_negotiation.cc:407] Connection header received
0619 06:58:31.827617 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.827619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.827661 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.827764 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:31.828292 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.828867 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.829611 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.829849 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.829930 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.832081 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:31.832084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:31.832087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:31.832109 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:31.833551 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.833987 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.833989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.833991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.834066 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:31.834303 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:31.834305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:31.834306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:31.834469 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:31.834568 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.834767 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.834857 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:58:31.838047 19220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:31.841252 19220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:31.856526 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.850398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.11.106.66:34551)
0619 06:58:31.851421 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.851432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.851453 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:58:31.851523 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.851528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.851711 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.851850 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:31.852096 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.853221 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.854780 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.855605 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.855731 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.855924 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.856256 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.856360 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":113,"threads_started":1}
I20260619 06:58:31.856580 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.849936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:34551 (local address 127.0.0.1:50128)
0619 06:58:31.850484 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:31.850500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:31.850598 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:31.851853 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:31.851857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:31.851876 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:31.851974 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.851981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.853348 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.853353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.854588 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.854622 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.855782 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.855785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.855990 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.856070 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 06:58:31.856202 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 06:58:31.856380 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":102,"thread_start_us":99,"threads_started":1}
I20260619 06:58:31.856580 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.849583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:37521 (local address 127.0.0.1:44584)
0619 06:58:31.850604 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:31.850618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:31.850733 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:31.851400 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:31.851406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:31.851432 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:31.851579 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.851585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.853249 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.853253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.854831 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.854837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.855797 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.855800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.855929 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.856071 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 06:58:31.856195 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 06:58:31.856380 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":169,"thread_start_us":89,"threads_started":1}
I20260619 06:58:31.857370 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.849725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.11.106.67:37521)
0619 06:58:31.850865 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.850881 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.850908 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:58:31.850987 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.850992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.851211 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.851741 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:31.851894 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.853130 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.854954 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.855680 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.855787 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.855927 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.857100 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.857227 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":110,"threads_started":1}
I20260619 06:58:31.857695 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.849667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:41247 (local address 127.0.0.1:57996)
0619 06:58:31.850351 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:31.850406 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:31.850535 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:31.851653 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:31.851655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:31.851669 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:31.851764 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.851769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.852985 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.852989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.854823 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:31.854835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.857238 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.857242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:31.857341 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.857366 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:58:31.857452 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:58:31.857542 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":74,"thread_start_us":98,"threads_started":1}
I20260619 06:58:31.858292 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:31.850805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.11.106.65:41247)
0619 06:58:31.851060 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:31.851073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:31.851092 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:58:31.851152 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:31.851155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:31.851370 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:31.851533 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:31.851910 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.852869 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.855990 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:31.856909 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:31.857015 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:31.857174 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:31.858067 (+ 893us) server_negotiation.cc:299] Negotiation successful
0619 06:58:31.858167 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
I20260619 06:58:31.859931 19498 tablet_service.cc:1467] Processing CreateTablet for tablet f013802029d74c24acbd5002029417e7 (DEFAULT_TABLE table=test-table [id=975fad860046468e9023f2d6520df46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:31.860242 19638 tablet_service.cc:1467] Processing CreateTablet for tablet f013802029d74c24acbd5002029417e7 (DEFAULT_TABLE table=test-table [id=975fad860046468e9023f2d6520df46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:31.861088 19498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f013802029d74c24acbd5002029417e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:31.860903 19358 tablet_service.cc:1467] Processing CreateTablet for tablet f013802029d74c24acbd5002029417e7 (DEFAULT_TABLE table=test-table [id=975fad860046468e9023f2d6520df46d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:31.861359 19638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f013802029d74c24acbd5002029417e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:31.861899 19358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f013802029d74c24acbd5002029417e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:31.868606 19732 tablet_bootstrap.cc:492] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: Bootstrap starting.
I20260619 06:58:31.869590 19733 tablet_bootstrap.cc:492] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: Bootstrap starting.
I20260619 06:58:31.870530 19734 tablet_bootstrap.cc:492] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: Bootstrap starting.
I20260619 06:58:31.871147 19732 tablet_bootstrap.cc:654] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:31.871740 19733 tablet_bootstrap.cc:654] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:31.872157 19732 log.cc:824] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:31.872866 19734 tablet_bootstrap.cc:654] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:31.873104 19733 log.cc:824] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:31.873682 19734 log.cc:824] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:31.874450 19732 tablet_bootstrap.cc:492] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: No bootstrap required, opened a new log
I20260619 06:58:31.874702 19732 ts_tablet_manager.cc:1397] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260619 06:58:31.874918 19733 tablet_bootstrap.cc:492] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: No bootstrap required, opened a new log
I20260619 06:58:31.875051 19733 ts_tablet_manager.cc:1397] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:58:31.875684 19734 tablet_bootstrap.cc:492] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: No bootstrap required, opened a new log
I20260619 06:58:31.875857 19734 ts_tablet_manager.cc:1397] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 06:58:31.877967 19733 raft_consensus.cc:348] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } }
I20260619 06:58:31.878180 19733 raft_consensus.cc:374] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:31.878223 19733 raft_consensus.cc:729] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a256df638b34b75a893c6e1056fc3c6, State: Initialized, Role: FOLLOWER
I20260619 06:58:31.878703 19733 consensus_queue.cc:260] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } }
I20260619 06:58:31.878804 19732 raft_consensus.cc:348] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } }
I20260619 06:58:31.879081 19732 raft_consensus.cc:374] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:31.879136 19732 raft_consensus.cc:729] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7735208f2f6a48898fb3da5767c9a7ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:31.879503 19424 heartbeater.cc:503] Master 127.11.106.126:36967 was elected leader, sending a full tablet report...
I20260619 06:58:31.879604 19732 consensus_queue.cc:260] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } }
I20260619 06:58:31.879714 19734 raft_consensus.cc:348] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } }
I20260619 06:58:31.879962 19734 raft_consensus.cc:374] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:31.880012 19734 raft_consensus.cc:729] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff64b894c21546f0b654955dab62df7b, State: Initialized, Role: FOLLOWER
I20260619 06:58:31.880710 19732 ts_tablet_manager.cc:1428] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20260619 06:58:31.880442 19734 consensus_queue.cc:260] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } }
I20260619 06:58:31.880972 19733 ts_tablet_manager.cc:1428] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 06:58:31.881078 19564 heartbeater.cc:503] Master 127.11.106.126:36967 was elected leader, sending a full tablet report...
I20260619 06:58:31.881767 19734 ts_tablet_manager.cc:1428] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260619 06:58:31.882309 19704 heartbeater.cc:503] Master 127.11.106.126:36967 was elected leader, sending a full tablet report...
W20260619 06:58:32.034447 19705 tablet.cc:2369] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:32.055166 19740 raft_consensus.cc:482] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:32.055415 19740 raft_consensus.cc:504] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } }
I20260619 06:58:32.056658 19740 leader_election.cc:288] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247), 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551)
I20260619 06:58:32.061209 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.057104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:50459 (local address 127.11.106.65:41247)
0619 06:58:32.057222 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.057226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.057243 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:58:32.057374 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.057377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.057428 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.057503 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:32.057902 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.058739 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.059989 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.060627 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.060778 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.060832 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.061015 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.061084 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:58:32.061241 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.056899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:41247 (local address 127.11.106.67:50459)
0619 06:58:32.057099 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.057115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.057229 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.057627 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.057630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.057650 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:32.057743 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.057748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.058854 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.058859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.059855 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.059866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.060764 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.060767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.060885 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.060910 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.060992 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.061074 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 06:58:32.061865 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.057455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:56001 (local address 127.11.106.66:34551)
0619 06:58:32.057577 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.057581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.057596 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:32.057647 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.057649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.057696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.057790 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:32.058587 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.059329 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.060489 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.061116 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.061223 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.061278 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.061687 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.061761 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:58:32.061879 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.056966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:34551 (local address 127.11.106.67:56001)
0619 06:58:32.057439 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.057453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.057530 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.057860 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.057865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.057882 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:32.058449 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.058460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.059458 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.059461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.060358 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.060367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.061481 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.061484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.061556 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.061573 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.061668 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.061745 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":124,"threads_started":1}
I20260619 06:58:32.062058 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f013802029d74c24acbd5002029417e7" candidate_uuid: "ff64b894c21546f0b654955dab62df7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a256df638b34b75a893c6e1056fc3c6" is_pre_election: true
I20260619 06:58:32.062366 19378 raft_consensus.cc:2454] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff64b894c21546f0b654955dab62df7b in term 0.
I20260619 06:58:32.062657 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f013802029d74c24acbd5002029417e7" candidate_uuid: "ff64b894c21546f0b654955dab62df7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7735208f2f6a48898fb3da5767c9a7ce" is_pre_election: true
I20260619 06:58:32.062860 19592 leader_election.cc:302] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a256df638b34b75a893c6e1056fc3c6, ff64b894c21546f0b654955dab62df7b; no voters:
I20260619 06:58:32.062950 19518 raft_consensus.cc:2454] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff64b894c21546f0b654955dab62df7b in term 0.
I20260619 06:58:32.063144 19740 raft_consensus.cc:2781] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:32.063222 19740 raft_consensus.cc:482] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:32.063270 19740 raft_consensus.cc:3037] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.064568 19740 raft_consensus.cc:504] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } }
I20260619 06:58:32.064922 19740 leader_election.cc:288] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [CANDIDATE]: Term 1 election: Requested vote from peers 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247), 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551)
I20260619 06:58:32.065316 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f013802029d74c24acbd5002029417e7" candidate_uuid: "ff64b894c21546f0b654955dab62df7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a256df638b34b75a893c6e1056fc3c6"
I20260619 06:58:32.065356 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f013802029d74c24acbd5002029417e7" candidate_uuid: "ff64b894c21546f0b654955dab62df7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7735208f2f6a48898fb3da5767c9a7ce"
I20260619 06:58:32.065472 19518 raft_consensus.cc:3037] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.065474 19378 raft_consensus.cc:3037] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.066751 19378 raft_consensus.cc:2454] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff64b894c21546f0b654955dab62df7b in term 1.
I20260619 06:58:32.066751 19518 raft_consensus.cc:2454] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff64b894c21546f0b654955dab62df7b in term 1.
I20260619 06:58:32.067050 19592 leader_election.cc:302] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a256df638b34b75a893c6e1056fc3c6, ff64b894c21546f0b654955dab62df7b; no voters:
I20260619 06:58:32.067248 19740 raft_consensus.cc:2781] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:32.067468 19740 raft_consensus.cc:686] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 1 LEADER]: Becoming Leader. State: Replica: ff64b894c21546f0b654955dab62df7b, State: Running, Role: LEADER
I20260619 06:58:32.067816 19740 consensus_queue.cc:237] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } }
W20260619 06:58:32.068751 19565 tablet.cc:2369] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:32.071285 19221 catalog_manager.cc:5495] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b reported cstate change: term changed from 0 to 1, leader changed from <none> to ff64b894c21546f0b654955dab62df7b (127.11.106.67). New cstate: current_term: 1 leader_uuid: "ff64b894c21546f0b654955dab62df7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:32.093974 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.090348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.11.106.67:37521)
0619 06:58:32.090478 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.090481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.090494 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:58:32.090570 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.090572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.090635 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.090740 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:32.090960 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.091474 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.092412 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.092735 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.092823 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.093128 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:32.093599 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:32.093698 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.093804 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.093858 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:58:32.100217 19518 raft_consensus.cc:1261] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Refusing update from remote peer ff64b894c21546f0b654955dab62df7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:32.100220 19378 raft_consensus.cc:1261] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Refusing update from remote peer ff64b894c21546f0b654955dab62df7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:32.101014 19742 consensus_queue.cc:1035] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:58:32.101140 19425 tablet.cc:2369] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:32.101225 19740 consensus_queue.cc:1035] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:32.107972 19745 mvcc.cc:205] Tried to move back new op lower bound from 7298467070354608128 to 7298467070232264704. Current Snapshot: MvccSnapshot[applied={T|T < 7298467070354608128}]
I20260619 06:58:32.118023 19220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:32.118251 19220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:32.121132 19220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29: Sending DeleteTablet for 3 replicas of tablet f013802029d74c24acbd5002029417e7
I20260619 06:58:32.121979 19638 tablet_service.cc:1514] Processing DeleteTablet for tablet f013802029d74c24acbd5002029417e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:32 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:44584
I20260619 06:58:32.122030 19498 tablet_service.cc:1514] Processing DeleteTablet for tablet f013802029d74c24acbd5002029417e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:32 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50128
I20260619 06:58:32.122044 19358 tablet_service.cc:1514] Processing DeleteTablet for tablet f013802029d74c24acbd5002029417e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:32 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:57996
I20260619 06:58:32.122555 19757 tablet_replica.cc:330] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: stopping tablet replica
I20260619 06:58:32.122710 19756 tablet_replica.cc:330] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: stopping tablet replica
I20260619 06:58:32.122929 19757 raft_consensus.cc:2229] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:32.123273 19757 raft_consensus.cc:2258] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:32.123435 19756 raft_consensus.cc:2229] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:32.123739 19220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:32.124248 19220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:32.125089 19757 ts_tablet_manager.cc:1905] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:32.125774 19756 raft_consensus.cc:2258] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:32.127235 19755 tablet_replica.cc:330] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: stopping tablet replica
I20260619 06:58:32.128247 19757 ts_tablet_manager.cc:1918] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:32.128373 19757 log.cc:1196] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/f013802029d74c24acbd5002029417e7
I20260619 06:58:32.128808 19757 ts_tablet_manager.cc:1939] T f013802029d74c24acbd5002029417e7 P 5a256df638b34b75a893c6e1056fc3c6: Deleting consensus metadata
I20260619 06:58:32.129931 19358 tablet_service.cc:1467] Processing CreateTablet for tablet b248610a58434bafb24605e76622e3aa (DEFAULT_TABLE table=test-table [id=77a98bfb386a4a72bdc95ea04695a8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:32.130218 19358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b248610a58434bafb24605e76622e3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:32.130300 19638 tablet_service.cc:1467] Processing CreateTablet for tablet b248610a58434bafb24605e76622e3aa (DEFAULT_TABLE table=test-table [id=77a98bfb386a4a72bdc95ea04695a8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:32.130442 19498 tablet_service.cc:1467] Processing CreateTablet for tablet b248610a58434bafb24605e76622e3aa (DEFAULT_TABLE table=test-table [id=77a98bfb386a4a72bdc95ea04695a8e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:32.130555 19638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b248610a58434bafb24605e76622e3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:32.130801 19498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b248610a58434bafb24605e76622e3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:32.133071 19733 tablet_bootstrap.cc:492] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: Bootstrap starting.
I20260619 06:58:32.133425 19732 tablet_bootstrap.cc:492] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: Bootstrap starting.
I20260619 06:58:32.133476 19734 tablet_bootstrap.cc:492] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: Bootstrap starting.
I20260619 06:58:32.134192 19733 tablet_bootstrap.cc:654] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:32.134477 19732 tablet_bootstrap.cc:654] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:32.134478 19734 tablet_bootstrap.cc:654] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:32.135104 19208 catalog_manager.cc:4841] TS 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247): tablet f013802029d74c24acbd5002029417e7 (table test-table [id=975fad860046468e9023f2d6520df46d]) successfully deleted
I20260619 06:58:32.135851 19755 raft_consensus.cc:2229] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:32.136057 19732 tablet_bootstrap.cc:492] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: No bootstrap required, opened a new log
I20260619 06:58:32.136132 19732 ts_tablet_manager.cc:1397] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:32.136479 19755 raft_consensus.cc:2258] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:32.136950 19732 raft_consensus.cc:348] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.137074 19732 raft_consensus.cc:374] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:32.137094 19756 ts_tablet_manager.cc:1905] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:32.137115 19732 raft_consensus.cc:729] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7735208f2f6a48898fb3da5767c9a7ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:32.137271 19732 consensus_queue.cc:260] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.137495 19734 tablet_bootstrap.cc:492] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: No bootstrap required, opened a new log
I20260619 06:58:32.137526 19732 ts_tablet_manager.cc:1428] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 06:58:32.137583 19734 ts_tablet_manager.cc:1397] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 06:58:32.138093 19734 raft_consensus.cc:348] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.138217 19734 raft_consensus.cc:374] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:32.138259 19734 raft_consensus.cc:729] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff64b894c21546f0b654955dab62df7b, State: Initialized, Role: FOLLOWER
I20260619 06:58:32.138413 19734 consensus_queue.cc:260] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.138769 19734 ts_tablet_manager.cc:1428] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 06:58:32.139211 19755 ts_tablet_manager.cc:1905] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:32.139997 19756 ts_tablet_manager.cc:1918] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:32.140141 19756 log.cc:1196] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/f013802029d74c24acbd5002029417e7
I20260619 06:58:32.140465 19756 ts_tablet_manager.cc:1939] T f013802029d74c24acbd5002029417e7 P 7735208f2f6a48898fb3da5767c9a7ce: Deleting consensus metadata
I20260619 06:58:32.141897 19755 ts_tablet_manager.cc:1918] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:32.141974 19755 log.cc:1196] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/f013802029d74c24acbd5002029417e7
I20260619 06:58:32.142263 19755 ts_tablet_manager.cc:1939] T f013802029d74c24acbd5002029417e7 P ff64b894c21546f0b654955dab62df7b: Deleting consensus metadata
I20260619 06:58:32.143366 19207 catalog_manager.cc:4841] TS ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521): tablet f013802029d74c24acbd5002029417e7 (table test-table [id=975fad860046468e9023f2d6520df46d]) successfully deleted
I20260619 06:58:32.144196 19733 tablet_bootstrap.cc:492] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: No bootstrap required, opened a new log
I20260619 06:58:32.144301 19733 ts_tablet_manager.cc:1397] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260619 06:58:32.144905 19205 catalog_manager.cc:4841] TS 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551): tablet f013802029d74c24acbd5002029417e7 (table test-table [id=975fad860046468e9023f2d6520df46d]) successfully deleted
I20260619 06:58:32.145126 19733 raft_consensus.cc:348] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.145267 19733 raft_consensus.cc:374] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:32.145301 19733 raft_consensus.cc:729] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a256df638b34b75a893c6e1056fc3c6, State: Initialized, Role: FOLLOWER
I20260619 06:58:32.145478 19733 consensus_queue.cc:260] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.145887 19733 ts_tablet_manager.cc:1428] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 06:58:32.166016 19738 raft_consensus.cc:482] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:32.166190 19738 raft_consensus.cc:504] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.167557 19738 leader_election.cc:288] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551), ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521)
I20260619 06:58:32.172365 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.167870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:48313 (local address 127.11.106.66:34551)
0619 06:58:32.168016 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.168022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.168129 (+ 107us) server_negotiation.cc:407] Connection header received
0619 06:58:32.168349 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.168353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.168411 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.168461 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:32.169100 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.169876 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.171099 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.171712 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.171843 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.171905 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.172141 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.172224 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:58:32.172405 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.167740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:34551 (local address 127.11.106.65:48313)
0619 06:58:32.168033 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.168050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.168211 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.168666 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.168668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.168694 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:32.168954 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.168962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.170014 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.170017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.170953 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.170963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.171870 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.171877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.172009 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.172031 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.172118 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.172196 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":43,"spinlock_wait_cycles":6912,"thread_start_us":69,"threads_started":1}
I20260619 06:58:32.173231 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b248610a58434bafb24605e76622e3aa" candidate_uuid: "5a256df638b34b75a893c6e1056fc3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7735208f2f6a48898fb3da5767c9a7ce" is_pre_election: true
I20260619 06:58:32.173338 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.168255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:60491 (local address 127.11.106.67:37521)
0619 06:58:32.168380 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.168385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.168401 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:32.168463 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.168466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.168516 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.168629 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:32.170054 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.170791 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.171991 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.172674 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.172802 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.172858 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.173165 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.173235 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:58:32.173345 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.167823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:37521 (local address 127.11.106.65:60491)
0619 06:58:32.168197 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.168210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.168288 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.168656 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.168659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.168689 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:32.168957 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.168962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.170937 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.170940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.171835 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.171844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.172911 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.172917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.173027 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.173046 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.173135 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.173202 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":104,"spinlock_wait_cycles":384,"thread_start_us":121,"threads_started":1}
I20260619 06:58:32.173413 19518 raft_consensus.cc:2454] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a256df638b34b75a893c6e1056fc3c6 in term 0.
I20260619 06:58:32.173820 19309 leader_election.cc:302] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a256df638b34b75a893c6e1056fc3c6, 7735208f2f6a48898fb3da5767c9a7ce; no voters:
I20260619 06:58:32.174100 19738 raft_consensus.cc:2781] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:32.174180 19738 raft_consensus.cc:482] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:32.174230 19738 raft_consensus.cc:3037] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.174636 19653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b248610a58434bafb24605e76622e3aa" candidate_uuid: "5a256df638b34b75a893c6e1056fc3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff64b894c21546f0b654955dab62df7b" is_pre_election: true
I20260619 06:58:32.174877 19653 raft_consensus.cc:2454] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a256df638b34b75a893c6e1056fc3c6 in term 0.
I20260619 06:58:32.176313 19738 raft_consensus.cc:504] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.176760 19738 leader_election.cc:288] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [CANDIDATE]: Term 1 election: Requested vote from peers 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551), ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521)
I20260619 06:58:32.177165 19653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b248610a58434bafb24605e76622e3aa" candidate_uuid: "5a256df638b34b75a893c6e1056fc3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff64b894c21546f0b654955dab62df7b"
I20260619 06:58:32.177155 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b248610a58434bafb24605e76622e3aa" candidate_uuid: "5a256df638b34b75a893c6e1056fc3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7735208f2f6a48898fb3da5767c9a7ce"
I20260619 06:58:32.177296 19518 raft_consensus.cc:3037] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.177299 19653 raft_consensus.cc:3037] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.178263 19653 raft_consensus.cc:2454] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a256df638b34b75a893c6e1056fc3c6 in term 1.
I20260619 06:58:32.178526 19518 raft_consensus.cc:2454] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a256df638b34b75a893c6e1056fc3c6 in term 1.
I20260619 06:58:32.178612 19311 leader_election.cc:302] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a256df638b34b75a893c6e1056fc3c6, ff64b894c21546f0b654955dab62df7b; no voters:
I20260619 06:58:32.178794 19738 raft_consensus.cc:2781] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:32.179040 19738 raft_consensus.cc:686] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5a256df638b34b75a893c6e1056fc3c6, State: Running, Role: LEADER
I20260619 06:58:32.179322 19738 consensus_queue.cc:237] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.182009 19221 catalog_manager.cc:5495] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65). New cstate: current_term: 1 leader_uuid: "5a256df638b34b75a893c6e1056fc3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } health_report { overall_health: UNKNOWN } } }
Jun 19 06:58:32 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852311, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.126@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:32.217960 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.206527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.11.106.126:36967)
0619 06:58:32.206661 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.206667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.206859 (+ 192us) server_negotiation.cc:407] Connection header received
0619 06:58:32.207025 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.207028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.207072 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.207159 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:32.207832 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.208726 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.209962 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.210559 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.210756 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.213998 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:32.214003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:32.214006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:32.214031 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:32.216120 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:32.216637 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:32.216641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:32.216643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:32.216723 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:32.217012 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:32.217015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:32.217017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:32.217202 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:32.217304 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.217610 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.217793 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":46}
I20260619 06:58:32.217960 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.206450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36967 (local address 127.0.0.1:42866)
0619 06:58:32.206769 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.206781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.206920 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.207177 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.207180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.207610 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:32.207699 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.207706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.208869 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.208873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.209809 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.209821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.210717 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.210721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.210888 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.211619 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:32.211636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:32.213757 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:32.216240 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:32.216251 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:32.216258 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:32.216494 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:32.216829 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:32.216832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:32.216834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:32.216917 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:32.217317 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:32.217327 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:32.217443 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.217563 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.217746 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":44,"thread_start_us":116,"threads_started":1}
I20260619 06:58:32.222097 19221 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:42866:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:58:32.222863 19221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:32.228286 19358 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2086b88872478dbfaed6ff397ccb02 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6c29cf5d2cd49628fa0c866d3956e44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:58:32.228513 19638 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2086b88872478dbfaed6ff397ccb02 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6c29cf5d2cd49628fa0c866d3956e44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:58:32.228533 19498 tablet_service.cc:1467] Processing CreateTablet for tablet 7c2086b88872478dbfaed6ff397ccb02 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6c29cf5d2cd49628fa0c866d3956e44]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:58:32.228739 19358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2086b88872478dbfaed6ff397ccb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:32.228914 19498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2086b88872478dbfaed6ff397ccb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:32.228933 19638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c2086b88872478dbfaed6ff397ccb02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:32.231724 19732 tablet_bootstrap.cc:492] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: Bootstrap starting.
I20260619 06:58:32.231777 19733 tablet_bootstrap.cc:492] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: Bootstrap starting.
I20260619 06:58:32.232714 19732 tablet_bootstrap.cc:654] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:32.232736 19733 tablet_bootstrap.cc:654] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:32.233419 19734 tablet_bootstrap.cc:492] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: Bootstrap starting.
I20260619 06:58:32.234215 19734 tablet_bootstrap.cc:654] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:32.234303 19732 tablet_bootstrap.cc:492] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: No bootstrap required, opened a new log
I20260619 06:58:32.234386 19732 ts_tablet_manager.cc:1397] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 06:58:32.234907 19732 raft_consensus.cc:348] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.235039 19732 raft_consensus.cc:374] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:32.235080 19732 raft_consensus.cc:729] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7735208f2f6a48898fb3da5767c9a7ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:32.235205 19732 consensus_queue.cc:260] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.235358 19733 tablet_bootstrap.cc:492] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: No bootstrap required, opened a new log
I20260619 06:58:32.235359 19734 tablet_bootstrap.cc:492] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: No bootstrap required, opened a new log
I20260619 06:58:32.235442 19734 ts_tablet_manager.cc:1397] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 06:58:32.235446 19733 ts_tablet_manager.cc:1397] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 06:58:32.235772 19739 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.235904 19739 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:32.235879 19733 raft_consensus.cc:348] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.235894 19734 raft_consensus.cc:348] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.235991 19733 raft_consensus.cc:374] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:32.236001 19734 raft_consensus.cc:374] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:32.236022 19733 raft_consensus.cc:729] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a256df638b34b75a893c6e1056fc3c6, State: Initialized, Role: FOLLOWER
I20260619 06:58:32.236034 19734 raft_consensus.cc:729] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff64b894c21546f0b654955dab62df7b, State: Initialized, Role: FOLLOWER
I20260619 06:58:32.236148 19734 consensus_queue.cc:260] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.236138 19733 consensus_queue.cc:260] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.236416 19734 ts_tablet_manager.cc:1428] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 06:58:32.236428 19733 ts_tablet_manager.cc:1428] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:58:32.236815 19732 ts_tablet_manager.cc:1428] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 06:58:32.236837 19750 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.236975 19750 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:32.239018 19738 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.239166 19738 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:32.480188 19750 raft_consensus.cc:482] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:32.480350 19750 raft_consensus.cc:504] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.480964 19750 leader_election.cc:288] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247), 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551)
I20260619 06:58:32.481590 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2086b88872478dbfaed6ff397ccb02" candidate_uuid: "ff64b894c21546f0b654955dab62df7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7735208f2f6a48898fb3da5767c9a7ce" is_pre_election: true
I20260619 06:58:32.481590 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2086b88872478dbfaed6ff397ccb02" candidate_uuid: "ff64b894c21546f0b654955dab62df7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a256df638b34b75a893c6e1056fc3c6" is_pre_election: true
I20260619 06:58:32.481791 19518 raft_consensus.cc:2454] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff64b894c21546f0b654955dab62df7b in term 0.
I20260619 06:58:32.481804 19378 raft_consensus.cc:2454] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff64b894c21546f0b654955dab62df7b in term 0.
I20260619 06:58:32.482218 19589 leader_election.cc:302] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7735208f2f6a48898fb3da5767c9a7ce, ff64b894c21546f0b654955dab62df7b; no voters:
I20260619 06:58:32.482409 19750 raft_consensus.cc:2781] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:32.482473 19750 raft_consensus.cc:482] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:32.482529 19750 raft_consensus.cc:3037] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.483696 19750 raft_consensus.cc:504] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.484115 19750 leader_election.cc:288] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [CANDIDATE]: Term 1 election: Requested vote from peers 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247), 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551)
I20260619 06:58:32.484606 19518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2086b88872478dbfaed6ff397ccb02" candidate_uuid: "ff64b894c21546f0b654955dab62df7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7735208f2f6a48898fb3da5767c9a7ce"
I20260619 06:58:32.484643 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c2086b88872478dbfaed6ff397ccb02" candidate_uuid: "ff64b894c21546f0b654955dab62df7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a256df638b34b75a893c6e1056fc3c6"
I20260619 06:58:32.484786 19518 raft_consensus.cc:3037] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.484809 19378 raft_consensus.cc:3037] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.485752 19518 raft_consensus.cc:2454] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff64b894c21546f0b654955dab62df7b in term 1.
I20260619 06:58:32.485857 19378 raft_consensus.cc:2454] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff64b894c21546f0b654955dab62df7b in term 1.
I20260619 06:58:32.486146 19589 leader_election.cc:302] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7735208f2f6a48898fb3da5767c9a7ce, ff64b894c21546f0b654955dab62df7b; no voters:
I20260619 06:58:32.486366 19750 raft_consensus.cc:2781] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:32.486488 19750 raft_consensus.cc:686] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [term 1 LEADER]: Becoming Leader. State: Replica: ff64b894c21546f0b654955dab62df7b, State: Running, Role: LEADER
I20260619 06:58:32.486677 19750 consensus_queue.cc:237] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.487432 19742 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: TxnStatusTablet state changed. Reason: New leader ff64b894c21546f0b654955dab62df7b. Latest consensus state: current_term: 1 leader_uuid: "ff64b894c21546f0b654955dab62df7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.487540 19742 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:58:32.487892 19775 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:58:32.488056 19775 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:58:32.488823 19220 catalog_manager.cc:5495] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b reported cstate change: term changed from 0 to 1, leader changed from <none> to ff64b894c21546f0b654955dab62df7b (127.11.106.67). New cstate: current_term: 1 leader_uuid: "ff64b894c21546f0b654955dab62df7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } health_report { overall_health: HEALTHY } } }
I20260619 06:58:32.537052 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.533400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:37521 (local address 127.0.0.1:32958)
0619 06:58:32.533559 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.533577 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.533709 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.534039 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.534042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.534048 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:32.534144 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.534149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.535134 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.535137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.535778 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.535788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.536337 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.536339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.536405 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.536453 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:32.536780 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:32.536795 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.536853 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.536930 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 06:58:32.537079 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.533661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.11.106.67:37521)
0619 06:58:32.533800 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.533805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.533821 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:32.533873 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.533876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.533927 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.534011 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:32.534244 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.535020 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.535886 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.536219 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.536335 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.536602 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:32.536689 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:32.536761 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.536886 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.536942 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:58:32.540063 19518 raft_consensus.cc:1261] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Refusing update from remote peer ff64b894c21546f0b654955dab62df7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:32.540095 19378 raft_consensus.cc:1261] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Refusing update from remote peer ff64b894c21546f0b654955dab62df7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:32.540565 19742 consensus_queue.cc:1035] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:32.540851 19750 consensus_queue.cc:1035] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:32.542192 19738 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: TxnStatusTablet state changed. Reason: New leader ff64b894c21546f0b654955dab62df7b. Latest consensus state: current_term: 1 leader_uuid: "ff64b894c21546f0b654955dab62df7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.542256 19739 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: TxnStatusTablet state changed. Reason: New leader ff64b894c21546f0b654955dab62df7b. Latest consensus state: current_term: 1 leader_uuid: "ff64b894c21546f0b654955dab62df7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.542336 19738 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:32.542362 19739 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:32.544088 19740 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff64b894c21546f0b654955dab62df7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.544234 19740 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:58:32.545491 19747 mvcc.cc:205] Tried to move back new op lower bound from 7298467072159637504 to 7298467071947071488. Current Snapshot: MvccSnapshot[applied={T|T < 7298467072159637504 or (T in {7298467072159637504})}]
I20260619 06:58:32.545498 19738 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff64b894c21546f0b654955dab62df7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.545938 19738 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P 5a256df638b34b75a893c6e1056fc3c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:32.547096 19750 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff64b894c21546f0b654955dab62df7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.547226 19750 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P ff64b894c21546f0b654955dab62df7b: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:58:32.548885 19739 tablet_replica.cc:439] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff64b894c21546f0b654955dab62df7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } } }
I20260619 06:58:32.549013 19739 tablet_replica.cc:442] T 7c2086b88872478dbfaed6ff397ccb02 P 7735208f2f6a48898fb3da5767c9a7ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:58:32.549028 19746 mvcc.cc:205] Tried to move back new op lower bound from 7298467072159637504 to 7298467071947071488. Current Snapshot: MvccSnapshot[applied={T|T < 7298467072159637504}]
I20260619 06:58:32.554188 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.549551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.11.106.65:41247)
0619 06:58:32.549656 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.549660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.549672 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:58:32.549752 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.549755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.549817 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.549920 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:32.550412 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.551175 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.552469 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.552772 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.552857 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.553230 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:32.553809 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:32.553914 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.554004 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.554065 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:58:32.566031 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.562225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:41249 (local address 127.11.106.67:37521)
0619 06:58:32.562402 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.562407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.562713 (+ 306us) server_negotiation.cc:407] Connection header received
0619 06:58:32.562852 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.562854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.562902 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.562960 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:32.563230 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.563714 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.565003 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.565233 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.565318 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.565638 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:32.565708 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:32.565759 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.565875 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.565921 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 06:58:32.566110 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.562082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:37521 (local address 127.11.106.65:41249)
0619 06:58:32.562612 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.562626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.562738 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.562995 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.562998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.563006 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:32.563131 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.563137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.563819 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.563822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.564864 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.564876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.565346 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.565350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.565446 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.565501 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:32.565785 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:32.565791 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.565858 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.565933 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":110,"threads_started":1}
I20260619 06:58:32.576017 19518 raft_consensus.cc:1261] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Refusing update from remote peer 5a256df638b34b75a893c6e1056fc3c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:32.576241 19653 raft_consensus.cc:1261] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Refusing update from remote peer 5a256df638b34b75a893c6e1056fc3c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:32.576557 19763 consensus_queue.cc:1035] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:32.576823 19738 consensus_queue.cc:1035] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:32.604182 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.599785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:41247 (local address 127.11.106.67:36029)
0619 06:58:32.600485 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.600499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.600660 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.600940 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.600943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.600951 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:32.601079 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.601084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.601977 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.601980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.602903 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.602915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.603485 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.603487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.603547 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.603589 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:32.603894 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:32.603903 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.603969 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.604053 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":90,"threads_started":1}
I20260619 06:58:32.604178 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.600129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:36029 (local address 127.11.106.65:41247)
0619 06:58:32.600256 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.600263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.600601 (+ 338us) server_negotiation.cc:407] Connection header received
0619 06:58:32.600778 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.600781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.600833 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.600917 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:32.601184 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.601847 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.603043 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.603333 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.603394 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.603726 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:32.603808 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:32.603890 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.603978 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.604029 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:58:32.635269 19221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:32.635489 19221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:32.638152 19221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29: Sending DeleteTablet for 3 replicas of tablet b248610a58434bafb24605e76622e3aa
I20260619 06:58:32.638839 19358 tablet_service.cc:1514] Processing DeleteTablet for tablet b248610a58434bafb24605e76622e3aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:32 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:57996
I20260619 06:58:32.639473 19638 tablet_service.cc:1514] Processing DeleteTablet for tablet b248610a58434bafb24605e76622e3aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:32 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:44584
I20260619 06:58:32.639501 19498 tablet_service.cc:1514] Processing DeleteTablet for tablet b248610a58434bafb24605e76622e3aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:32 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50128
I20260619 06:58:32.639663 19755 tablet_replica.cc:330] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: stopping tablet replica
I20260619 06:58:32.639695 19756 tablet_replica.cc:330] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: stopping tablet replica
I20260619 06:58:32.639782 19755 raft_consensus.cc:2229] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:32.639812 19756 raft_consensus.cc:2229] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:32.639896 19755 raft_consensus.cc:2258] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:32.639927 19756 raft_consensus.cc:2258] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:32.640961 19755 ts_tablet_manager.cc:1905] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:32.643442 19790 tablet_replica.cc:330] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: stopping tablet replica
I20260619 06:58:32.643561 19790 raft_consensus.cc:2229] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:32.643860 19790 raft_consensus.cc:2258] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:32.644007 19755 ts_tablet_manager.cc:1918] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:32.644078 19755 log.cc:1196] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/b248610a58434bafb24605e76622e3aa
I20260619 06:58:32.644351 19755 ts_tablet_manager.cc:1939] T b248610a58434bafb24605e76622e3aa P ff64b894c21546f0b654955dab62df7b: Deleting consensus metadata
I20260619 06:58:32.644507 19790 ts_tablet_manager.cc:1905] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:32.646347 19207 catalog_manager.cc:4841] TS ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521): tablet b248610a58434bafb24605e76622e3aa (table test-table [id=77a98bfb386a4a72bdc95ea04695a8e6]) successfully deleted
I20260619 06:58:32.647353 19790 ts_tablet_manager.cc:1918] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:32.647372 19756 ts_tablet_manager.cc:1905] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:32.647428 19790 log.cc:1196] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/b248610a58434bafb24605e76622e3aa
I20260619 06:58:32.647683 19790 ts_tablet_manager.cc:1939] T b248610a58434bafb24605e76622e3aa P 5a256df638b34b75a893c6e1056fc3c6: Deleting consensus metadata
I20260619 06:58:32.648406 19208 catalog_manager.cc:4841] TS 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247): tablet b248610a58434bafb24605e76622e3aa (table test-table [id=77a98bfb386a4a72bdc95ea04695a8e6]) successfully deleted
I20260619 06:58:32.650312 19756 ts_tablet_manager.cc:1918] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:32.650393 19756 log.cc:1196] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/b248610a58434bafb24605e76622e3aa
I20260619 06:58:32.650695 19756 ts_tablet_manager.cc:1939] T b248610a58434bafb24605e76622e3aa P 7735208f2f6a48898fb3da5767c9a7ce: Deleting consensus metadata
I20260619 06:58:32.651615 19205 catalog_manager.cc:4841] TS 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551): tablet b248610a58434bafb24605e76622e3aa (table test-table [id=77a98bfb386a4a72bdc95ea04695a8e6]) successfully deleted
Jun 19 06:58:32 dist-test-slave-10d0 krb5kdc[19176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:58:32 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:32 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:58:32.679136 19270 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:32.671356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.11.106.126:36967)
0619 06:58:32.671508 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.671515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.671757 (+ 242us) server_negotiation.cc:407] Connection header received
0619 06:58:32.671909 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.671911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.671993 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.672069 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:32.673181 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.673743 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.675048 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.675409 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.675497 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.678982 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45676: BlockingRecv error: recv got EOF from 127.0.0.1:45676 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:58:32 dist-test-slave-10d0 krb5kdc[19176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM
I20260619 06:58:32.690467 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.680763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.11.106.126:36967)
0619 06:58:32.680891 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.680894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.681113 (+ 219us) server_negotiation.cc:407] Connection header received
0619 06:58:32.681205 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.681208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.681263 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.681360 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:32.681920 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.682752 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.683824 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.684195 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.684309 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.686751 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:32.686756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:32.686759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:32.686792 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:32.688442 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:32.688881 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:32.688885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:32.688887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:32.689034 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:32.689479 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:32.689483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:32.689485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:32.689763 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:32.689940 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.690102 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.690278 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:58:32.693517 19221 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:32.694036 19221 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:32.699484 19358 tablet_service.cc:1467] Processing CreateTablet for tablet ff1025769bdd4af8bb1bebbc5fa772c9 (DEFAULT_TABLE table=test-table [id=b2a82120cdc942248517a52bf57ccb63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:32.699608 19638 tablet_service.cc:1467] Processing CreateTablet for tablet ff1025769bdd4af8bb1bebbc5fa772c9 (DEFAULT_TABLE table=test-table [id=b2a82120cdc942248517a52bf57ccb63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:32.699716 19498 tablet_service.cc:1467] Processing CreateTablet for tablet ff1025769bdd4af8bb1bebbc5fa772c9 (DEFAULT_TABLE table=test-table [id=b2a82120cdc942248517a52bf57ccb63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:32.699813 19358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1025769bdd4af8bb1bebbc5fa772c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:32.699903 19638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1025769bdd4af8bb1bebbc5fa772c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:32.700006 19498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1025769bdd4af8bb1bebbc5fa772c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:32.702354 19733 tablet_bootstrap.cc:492] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: Bootstrap starting.
I20260619 06:58:32.702551 19734 tablet_bootstrap.cc:492] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: Bootstrap starting.
I20260619 06:58:32.703330 19733 tablet_bootstrap.cc:654] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:32.703466 19734 tablet_bootstrap.cc:654] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:32.704088 19732 tablet_bootstrap.cc:492] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: Bootstrap starting.
I20260619 06:58:32.704954 19734 tablet_bootstrap.cc:492] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: No bootstrap required, opened a new log
I20260619 06:58:32.704954 19733 tablet_bootstrap.cc:492] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: No bootstrap required, opened a new log
I20260619 06:58:32.705036 19734 ts_tablet_manager.cc:1397] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 06:58:32.705036 19733 ts_tablet_manager.cc:1397] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:32.705056 19732 tablet_bootstrap.cc:654] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:32.705473 19733 raft_consensus.cc:348] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.705459 19734 raft_consensus.cc:348] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.705574 19734 raft_consensus.cc:374] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:32.705574 19733 raft_consensus.cc:374] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:32.705608 19734 raft_consensus.cc:729] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff64b894c21546f0b654955dab62df7b, State: Initialized, Role: FOLLOWER
I20260619 06:58:32.705608 19733 raft_consensus.cc:729] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a256df638b34b75a893c6e1056fc3c6, State: Initialized, Role: FOLLOWER
I20260619 06:58:32.705734 19734 consensus_queue.cc:260] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.705734 19733 consensus_queue.cc:260] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.705996 19734 ts_tablet_manager.cc:1428] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:58:32.705996 19733 ts_tablet_manager.cc:1428] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:32.706245 19732 tablet_bootstrap.cc:492] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: No bootstrap required, opened a new log
I20260619 06:58:32.706337 19732 ts_tablet_manager.cc:1397] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:32.706799 19732 raft_consensus.cc:348] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.706904 19732 raft_consensus.cc:374] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:32.706941 19732 raft_consensus.cc:729] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7735208f2f6a48898fb3da5767c9a7ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:32.707065 19732 consensus_queue.cc:260] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.707333 19732 ts_tablet_manager.cc:1428] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:32.973621 19739 raft_consensus.cc:482] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:32.973796 19739 raft_consensus.cc:504] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.975072 19739 leader_election.cc:288] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247), ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521)
I20260619 06:58:32.979732 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.975417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:51971 (local address 127.11.106.65:41247)
0619 06:58:32.975545 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.975551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.975783 (+ 232us) server_negotiation.cc:407] Connection header received
0619 06:58:32.975906 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.975910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.975970 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.976050 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:32.976384 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.977174 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.978416 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.979148 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.979283 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.979351 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.979528 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.979590 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:58:32.979775 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.975344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:41247 (local address 127.11.106.66:51971)
0619 06:58:32.975682 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.975697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.975799 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.976074 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.976077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.976095 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:32.976252 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.976259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.977301 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.977305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.978259 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.978269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.979274 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.979278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.979394 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.979421 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.979501 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.979568 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":86,"threads_started":1}
I20260619 06:58:32.980433 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.975480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:33619 (local address 127.11.106.67:37521)
0619 06:58:32.975621 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:32.975626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:32.975937 (+ 311us) server_negotiation.cc:407] Connection header received
0619 06:58:32.976075 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:32.976078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:32.976128 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:32.976211 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:32.976918 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.977581 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.979020 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.979636 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.979775 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.979837 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:32.980222 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 06:58:32.980301 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:58:32.980441 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:32.975372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:37521 (local address 127.11.106.66:33619)
0619 06:58:32.975829 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:32.975845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:32.975949 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:32.976239 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:32.976242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:32.976257 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:32.976799 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.976809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.977801 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.977804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.978889 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:32.978901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:32.979768 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:32.979774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:32.980068 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:32.980101 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:58:32.980188 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:58:32.980253 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":65,"threads_started":1}
I20260619 06:58:32.980453 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1025769bdd4af8bb1bebbc5fa772c9" candidate_uuid: "7735208f2f6a48898fb3da5767c9a7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a256df638b34b75a893c6e1056fc3c6" is_pre_election: true
I20260619 06:58:32.980640 19378 raft_consensus.cc:2454] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7735208f2f6a48898fb3da5767c9a7ce in term 0.
I20260619 06:58:32.980959 19653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1025769bdd4af8bb1bebbc5fa772c9" candidate_uuid: "7735208f2f6a48898fb3da5767c9a7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff64b894c21546f0b654955dab62df7b" is_pre_election: true
I20260619 06:58:32.981035 19452 leader_election.cc:302] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a256df638b34b75a893c6e1056fc3c6, 7735208f2f6a48898fb3da5767c9a7ce; no voters:
I20260619 06:58:32.981122 19653 raft_consensus.cc:2454] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7735208f2f6a48898fb3da5767c9a7ce in term 0.
I20260619 06:58:32.981374 19739 raft_consensus.cc:2781] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:32.981462 19739 raft_consensus.cc:482] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:32.981503 19739 raft_consensus.cc:3037] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.982524 19739 raft_consensus.cc:504] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.982975 19739 leader_election.cc:288] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [CANDIDATE]: Term 1 election: Requested vote from peers 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247), ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521)
I20260619 06:58:32.983297 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1025769bdd4af8bb1bebbc5fa772c9" candidate_uuid: "7735208f2f6a48898fb3da5767c9a7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a256df638b34b75a893c6e1056fc3c6"
I20260619 06:58:32.983336 19653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1025769bdd4af8bb1bebbc5fa772c9" candidate_uuid: "7735208f2f6a48898fb3da5767c9a7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff64b894c21546f0b654955dab62df7b"
I20260619 06:58:32.983428 19378 raft_consensus.cc:3037] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.983448 19653 raft_consensus.cc:3037] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:32.984467 19378 raft_consensus.cc:2454] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7735208f2f6a48898fb3da5767c9a7ce in term 1.
I20260619 06:58:32.984470 19653 raft_consensus.cc:2454] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7735208f2f6a48898fb3da5767c9a7ce in term 1.
I20260619 06:58:32.984771 19452 leader_election.cc:302] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a256df638b34b75a893c6e1056fc3c6, 7735208f2f6a48898fb3da5767c9a7ce; no voters:
I20260619 06:58:32.984949 19739 raft_consensus.cc:2781] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:32.985127 19739 raft_consensus.cc:686] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 LEADER]: Becoming Leader. State: Replica: 7735208f2f6a48898fb3da5767c9a7ce, State: Running, Role: LEADER
I20260619 06:58:32.985399 19739 consensus_queue.cc:237] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:32.987681 19221 catalog_manager.cc:5495] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66). New cstate: current_term: 1 leader_uuid: "7735208f2f6a48898fb3da5767c9a7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:33.006359 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:33.002169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.11.106.66:34551)
0619 06:58:33.002482 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:33.002489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:33.002508 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:58:33.002609 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:33.002616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:33.002727 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:33.002873 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:33.003143 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.003729 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.004604 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.004917 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.005008 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:33.005302 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:33.005830 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:33.005932 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:33.006062 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 06:58:33.006124 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":154,"threads_started":1}
I20260619 06:58:33.009298 19653 raft_consensus.cc:1261] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Refusing update from remote peer 7735208f2f6a48898fb3da5767c9a7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:33.009312 19378 raft_consensus.cc:1261] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Refusing update from remote peer 7735208f2f6a48898fb3da5767c9a7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:33.009828 19808 consensus_queue.cc:1035] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:33.010001 19739 consensus_queue.cc:1035] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:33.021561 19220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:33.021750 19220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:33.024374 19220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29: Sending DeleteTablet for 3 replicas of tablet ff1025769bdd4af8bb1bebbc5fa772c9
I20260619 06:58:33.025133 19498 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1025769bdd4af8bb1bebbc5fa772c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:33 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50128
I20260619 06:58:33.025331 19358 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1025769bdd4af8bb1bebbc5fa772c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:33 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:57996
I20260619 06:58:33.025381 19497 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1025769bdd4af8bb1bebbc5fa772c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:33 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50128
I20260619 06:58:33.025568 19638 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1025769bdd4af8bb1bebbc5fa772c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:33 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:44584
I20260619 06:58:33.025731 19756 tablet_replica.cc:330] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: stopping tablet replica
I20260619 06:58:33.025775 19755 tablet_replica.cc:330] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: stopping tablet replica
I20260619 06:58:33.025830 19756 raft_consensus.cc:2229] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:33.025909 19755 raft_consensus.cc:2229] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:33.026077 19755 raft_consensus.cc:2258] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:33.026165 19756 raft_consensus.cc:2258] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:33.026383 19790 tablet_replica.cc:330] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: stopping tablet replica
I20260619 06:58:33.026512 19790 raft_consensus.cc:2229] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:33.026654 19790 raft_consensus.cc:2258] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:33.026928 19756 ts_tablet_manager.cc:1905] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:33.027711 19790 ts_tablet_manager.cc:1905] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:33.029716 19756 ts_tablet_manager.cc:1918] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:33.029803 19756 log.cc:1196] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/ff1025769bdd4af8bb1bebbc5fa772c9
I20260619 06:58:33.029807 19790 ts_tablet_manager.cc:1918] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:33.029891 19790 log.cc:1196] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/ff1025769bdd4af8bb1bebbc5fa772c9
I20260619 06:58:33.030128 19790 ts_tablet_manager.cc:1939] T ff1025769bdd4af8bb1bebbc5fa772c9 P 5a256df638b34b75a893c6e1056fc3c6: Deleting consensus metadata
I20260619 06:58:33.030303 19756 ts_tablet_manager.cc:1939] T ff1025769bdd4af8bb1bebbc5fa772c9 P 7735208f2f6a48898fb3da5767c9a7ce: Deleting consensus metadata
I20260619 06:58:33.030951 19208 catalog_manager.cc:4841] TS 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247): tablet ff1025769bdd4af8bb1bebbc5fa772c9 (table test-table [id=b2a82120cdc942248517a52bf57ccb63]) successfully deleted
I20260619 06:58:33.031069 19205 catalog_manager.cc:4841] TS 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551): tablet ff1025769bdd4af8bb1bebbc5fa772c9 (table test-table [id=b2a82120cdc942248517a52bf57ccb63]) successfully deleted
W20260619 06:58:33.031481 19205 catalog_manager.cc:4808] TS 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551): delete failed for tablet ff1025769bdd4af8bb1bebbc5fa772c9 because the tablet was not found. No further retry: Not found: Tablet not found: ff1025769bdd4af8bb1bebbc5fa772c9
I20260619 06:58:33.031473 19220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:33.032003 19220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:33.032261 19755 ts_tablet_manager.cc:1905] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:33.034631 19755 ts_tablet_manager.cc:1918] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:33.034732 19755 log.cc:1196] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/ff1025769bdd4af8bb1bebbc5fa772c9
I20260619 06:58:33.034969 19755 ts_tablet_manager.cc:1939] T ff1025769bdd4af8bb1bebbc5fa772c9 P ff64b894c21546f0b654955dab62df7b: Deleting consensus metadata
I20260619 06:58:33.035777 19207 catalog_manager.cc:4841] TS ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521): tablet ff1025769bdd4af8bb1bebbc5fa772c9 (table test-table [id=b2a82120cdc942248517a52bf57ccb63]) successfully deleted
I20260619 06:58:33.037958 19638 tablet_service.cc:1467] Processing CreateTablet for tablet 86501d67cc1c405aa68f4639af981f8c (DEFAULT_TABLE table=test-table [id=b954bd06c54344278c866ab277b76161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:33.037959 19497 tablet_service.cc:1467] Processing CreateTablet for tablet 86501d67cc1c405aa68f4639af981f8c (DEFAULT_TABLE table=test-table [id=b954bd06c54344278c866ab277b76161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:33.038251 19638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86501d67cc1c405aa68f4639af981f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:33.038259 19497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86501d67cc1c405aa68f4639af981f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:33.038239 19358 tablet_service.cc:1467] Processing CreateTablet for tablet 86501d67cc1c405aa68f4639af981f8c (DEFAULT_TABLE table=test-table [id=b954bd06c54344278c866ab277b76161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:33.038525 19358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86501d67cc1c405aa68f4639af981f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:33.040808 19734 tablet_bootstrap.cc:492] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: Bootstrap starting.
I20260619 06:58:33.040858 19732 tablet_bootstrap.cc:492] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: Bootstrap starting.
I20260619 06:58:33.041738 19734 tablet_bootstrap.cc:654] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:33.041791 19732 tablet_bootstrap.cc:654] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:33.042306 19733 tablet_bootstrap.cc:492] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6: Bootstrap starting.
I20260619 06:58:33.043154 19732 tablet_bootstrap.cc:492] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: No bootstrap required, opened a new log
I20260619 06:58:33.043236 19733 tablet_bootstrap.cc:654] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:33.043241 19732 ts_tablet_manager.cc:1397] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 06:58:33.043723 19732 raft_consensus.cc:348] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:33.043843 19732 raft_consensus.cc:374] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:33.043877 19732 raft_consensus.cc:729] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7735208f2f6a48898fb3da5767c9a7ce, State: Initialized, Role: FOLLOWER
I20260619 06:58:33.044029 19732 consensus_queue.cc:260] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:33.044282 19732 ts_tablet_manager.cc:1428] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 06:58:33.044335 19733 tablet_bootstrap.cc:492] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6: No bootstrap required, opened a new log
I20260619 06:58:33.044412 19733 ts_tablet_manager.cc:1397] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:33.044864 19733 raft_consensus.cc:348] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:33.044986 19733 raft_consensus.cc:374] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:33.045024 19733 raft_consensus.cc:729] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a256df638b34b75a893c6e1056fc3c6, State: Initialized, Role: FOLLOWER
I20260619 06:58:33.045151 19733 consensus_queue.cc:260] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:33.045426 19733 ts_tablet_manager.cc:1428] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 06:58:33.045765 19734 tablet_bootstrap.cc:492] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: No bootstrap required, opened a new log
I20260619 06:58:33.045863 19734 ts_tablet_manager.cc:1397] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260619 06:58:33.046295 19734 raft_consensus.cc:348] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:33.046419 19734 raft_consensus.cc:374] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:33.046452 19734 raft_consensus.cc:729] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff64b894c21546f0b654955dab62df7b, State: Initialized, Role: FOLLOWER
I20260619 06:58:33.046595 19734 consensus_queue.cc:260] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:33.046875 19734 ts_tablet_manager.cc:1428] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 06:58:33.047885 19814 raft_consensus.cc:482] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:33.047974 19814 raft_consensus.cc:504] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:33.048470 19814 leader_election.cc:288] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247), ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521)
I20260619 06:58:33.048679 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86501d67cc1c405aa68f4639af981f8c" candidate_uuid: "7735208f2f6a48898fb3da5767c9a7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a256df638b34b75a893c6e1056fc3c6" is_pre_election: true
I20260619 06:58:33.048679 19653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86501d67cc1c405aa68f4639af981f8c" candidate_uuid: "7735208f2f6a48898fb3da5767c9a7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff64b894c21546f0b654955dab62df7b" is_pre_election: true
I20260619 06:58:33.048800 19378 raft_consensus.cc:2454] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7735208f2f6a48898fb3da5767c9a7ce in term 0.
I20260619 06:58:33.048800 19653 raft_consensus.cc:2454] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7735208f2f6a48898fb3da5767c9a7ce in term 0.
I20260619 06:58:33.049134 19451 leader_election.cc:302] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7735208f2f6a48898fb3da5767c9a7ce, ff64b894c21546f0b654955dab62df7b; no voters:
I20260619 06:58:33.049345 19814 raft_consensus.cc:2781] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:33.049400 19814 raft_consensus.cc:482] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:33.049443 19814 raft_consensus.cc:3037] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:33.050611 19814 raft_consensus.cc:504] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:33.051244 19814 leader_election.cc:288] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [CANDIDATE]: Term 1 election: Requested vote from peers 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247), ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521)
I20260619 06:58:33.051306 19653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86501d67cc1c405aa68f4639af981f8c" candidate_uuid: "7735208f2f6a48898fb3da5767c9a7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff64b894c21546f0b654955dab62df7b"
I20260619 06:58:33.051415 19653 raft_consensus.cc:3037] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:33.051561 19378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86501d67cc1c405aa68f4639af981f8c" candidate_uuid: "7735208f2f6a48898fb3da5767c9a7ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a256df638b34b75a893c6e1056fc3c6"
I20260619 06:58:33.051663 19378 raft_consensus.cc:3037] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:33.052328 19653 raft_consensus.cc:2454] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7735208f2f6a48898fb3da5767c9a7ce in term 1.
I20260619 06:58:33.052851 19378 raft_consensus.cc:2454] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7735208f2f6a48898fb3da5767c9a7ce in term 1.
I20260619 06:58:33.052903 19451 leader_election.cc:302] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7735208f2f6a48898fb3da5767c9a7ce, ff64b894c21546f0b654955dab62df7b; no voters:
I20260619 06:58:33.053304 19814 raft_consensus.cc:2781] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:33.053514 19814 raft_consensus.cc:686] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 1 LEADER]: Becoming Leader. State: Replica: 7735208f2f6a48898fb3da5767c9a7ce, State: Running, Role: LEADER
I20260619 06:58:33.053678 19814 consensus_queue.cc:237] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } }
I20260619 06:58:33.055373 19221 catalog_manager.cc:5495] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66). New cstate: current_term: 1 leader_uuid: "7735208f2f6a48898fb3da5767c9a7ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7735208f2f6a48898fb3da5767c9a7ce" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 } health_report { overall_health: UNKNOWN } } }
I20260619 06:58:33.081132 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:33.077490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:57007 (local address 127.11.106.67:37521)
0619 06:58:33.077621 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:33.077627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:33.077789 (+ 162us) server_negotiation.cc:407] Connection header received
0619 06:58:33.077885 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:33.077888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:33.077944 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:33.078030 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:33.078255 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.079061 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.079922 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.080270 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.080371 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:33.080695 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:33.080780 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:33.080855 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:33.080938 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 06:58:33.080992 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:58:33.083877 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:33.077391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:37521 (local address 127.11.106.66:57007)
0619 06:58:33.077700 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:33.077715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:33.077807 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:33.078042 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:33.078045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:33.078051 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:33.078155 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:33.078159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.079175 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.079177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:33.079804 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:33.079813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.080394 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.080396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:33.080474 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:33.080520 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:33.080870 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:33.080876 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:33.080928 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 06:58:33.080992 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
I20260619 06:58:33.091440 19653 raft_consensus.cc:1261] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Refusing update from remote peer 7735208f2f6a48898fb3da5767c9a7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:33.091614 19378 raft_consensus.cc:1261] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Refusing update from remote peer 7735208f2f6a48898fb3da5767c9a7ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:33.092008 19814 consensus_queue.cc:1035] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a256df638b34b75a893c6e1056fc3c6" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:33.092248 19739 consensus_queue.cc:1035] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff64b894c21546f0b654955dab62df7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:33.112697 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:33.108775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:41753 (local address 127.11.106.66:34551)
0619 06:58:33.108888 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:33.108893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:33.109114 (+ 221us) server_negotiation.cc:407] Connection header received
0619 06:58:33.109292 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:33.109295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:33.109343 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:33.109388 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:33.109762 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.110290 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.111436 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.111759 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.111827 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:33.112175 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:33.112260 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:33.112302 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:33.112458 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 06:58:33.112510 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 06:58:33.112721 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:33.108705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:34551 (local address 127.11.106.67:41753)
0619 06:58:33.109019 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:33.109052 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:33.109178 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:33.109531 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:33.109534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:33.109542 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:58:33.109642 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:33.109648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.110399 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.110403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:33.111293 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:33.111304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:33.111936 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:33.111940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:33.112026 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:33.112055 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:58:33.112361 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:58:33.112366 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:58:33.112430 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 06:58:33.112516 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":67,"threads_started":1}
I20260619 06:58:33.137390 19221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:33.137567 19221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:33.139863 19221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a94c8f6cc854c4094bcf884cc875e29: Sending DeleteTablet for 3 replicas of tablet 86501d67cc1c405aa68f4639af981f8c
I20260619 06:58:33.140475 19358 tablet_service.cc:1514] Processing DeleteTablet for tablet 86501d67cc1c405aa68f4639af981f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:33 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:57996
I20260619 06:58:33.140638 19497 tablet_service.cc:1514] Processing DeleteTablet for tablet 86501d67cc1c405aa68f4639af981f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:33 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50128
I20260619 06:58:33.140710 19790 tablet_replica.cc:330] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6: stopping tablet replica
I20260619 06:58:33.140822 19756 tablet_replica.cc:330] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: stopping tablet replica
I20260619 06:58:33.140833 19790 raft_consensus.cc:2229] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:33.140836 19638 tablet_service.cc:1514] Processing DeleteTablet for tablet 86501d67cc1c405aa68f4639af981f8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:33 UTC) from {username='oryx', principal='oryx/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:44584
I20260619 06:58:33.140955 19790 raft_consensus.cc:2258] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:33.140951 19756 raft_consensus.cc:2229] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:33.141000 19755 tablet_replica.cc:330] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: stopping tablet replica
I20260619 06:58:33.141099 19755 raft_consensus.cc:2229] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:33.141146 19756 raft_consensus.cc:2258] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:33.141208 19755 raft_consensus.cc:2258] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:33.141919 19790 ts_tablet_manager.cc:1905] T 86501d67cc1c405aa68f4639af981f8c P 5a256df638b34b75a893c6e1056fc3c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:33.141918 19756 ts_tablet_manager.cc:1905] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:33.142746 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 19293
I20260619 06:58:33.144228 19756 ts_tablet_manager.cc:1918] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:33.144352 19756 log.cc:1196] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/86501d67cc1c405aa68f4639af981f8c
I20260619 06:58:33.145596 19755 ts_tablet_manager.cc:1905] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:33.147637 19755 ts_tablet_manager.cc:1918] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:58:33.147742 19755 log.cc:1196] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/86501d67cc1c405aa68f4639af981f8c
I20260619 06:58:33.148010 19755 ts_tablet_manager.cc:1939] T 86501d67cc1c405aa68f4639af981f8c P ff64b894c21546f0b654955dab62df7b: Deleting consensus metadata
I20260619 06:58:33.148753 19207 catalog_manager.cc:4841] TS ff64b894c21546f0b654955dab62df7b (127.11.106.67:37521): tablet 86501d67cc1c405aa68f4639af981f8c (table test-table [id=b954bd06c54344278c866ab277b76161]) successfully deleted
I20260619 06:58:33.152752 19756 ts_tablet_manager.cc:1939] T 86501d67cc1c405aa68f4639af981f8c P 7735208f2f6a48898fb3da5767c9a7ce: Deleting consensus metadata
I20260619 06:58:33.153643 19205 catalog_manager.cc:4841] TS 7735208f2f6a48898fb3da5767c9a7ce (127.11.106.66:34551): tablet 86501d67cc1c405aa68f4639af981f8c (table test-table [id=b954bd06c54344278c866ab277b76161]) successfully deleted
W20260619 06:58:33.156392 19208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:41247 (error 108)
I20260619 06:58:33.157505 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 19433
I20260619 06:58:33.158660 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:33.157329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:41247 (local address 127.0.0.1:58358)
0619 06:58:33.158424 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:33.158517 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":63,"threads_started":1}
W20260619 06:58:33.158826 19208 catalog_manager.cc:4584] TS 5a256df638b34b75a893c6e1056fc3c6 (127.11.106.65:41247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86501d67cc1c405aa68f4639af981f8c: Network error: Client connection negotiation failed: client connection to 127.11.106.65:41247: connect: Connection refused (error 111)
I20260619 06:58:33.163935 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 19573
I20260619 06:58:33.171375 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 19189
2026-06-19T06:58:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3519 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:33 dist-test-slave-10d0 krb5kdc[19831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:33 dist-test-slave-10d0 krb5kdc[19831](info): set up 2 sockets
Jun 19 06:58:33 dist-test-slave-10d0 krb5kdc[19831](info): commencing operation
krb5kdc: starting...
W20260619 06:58:34.225977 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:34 dist-test-slave-10d0 krb5kdc[19831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:34.351508 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36167
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:36167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:34.466786 19844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:34.467154 19844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:34.467249 19844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:34.471463 19844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:34.471581 19844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:34.471628 19844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:34.471693 19844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:34.471736 19844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:34.477186 19844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:36167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:34.478391 19844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:34.479554 19844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:34.488691 19849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:34.488732 19852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:34.488924 19844 server_base.cc:1034] running on GCE node
W20260619 06:58:34.488740 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:34.489585 19844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:34.490767 19844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:34.491966 19844 hybrid_clock.cc:648] HybridClock initialized: now 1781852314491934 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T06:58:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1326 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:34 dist-test-slave-10d0 krb5kdc[19857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:34 dist-test-slave-10d0 krb5kdc[19857](info): set up 2 sockets
Jun 19 06:58:34 dist-test-slave-10d0 krb5kdc[19857](info): commencing operation
krb5kdc: starting...
W20260619 06:58:35.533833 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:35 dist-test-slave-10d0 krb5kdc[19857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:35.653687 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36423
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:35.763314 19870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:35.763618 19870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:35.763715 19870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:35.767748 19870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:35.767845 19870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:35.767882 19870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:35.767936 19870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:35.767977 19870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:35.773301 19870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:35.774405 19870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:35.775476 19870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:35.784036 19876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:35.784025 19878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:35.784025 19875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:35.784157 19870 server_base.cc:1034] running on GCE node
I20260619 06:58:35.785044 19870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:35.786164 19870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:35.787321 19870 hybrid_clock.cc:648] HybridClock initialized: now 1781852315787303 us; error 31 us; skew 500 ppm
Jun 19 06:58:35 dist-test-slave-10d0 krb5kdc[19857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852315, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:35.790108 19870 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:35.790944 19870 webserver.cc:460] Webserver started at http://127.11.106.126:43817/ using document root <none> and password file <none>
I20260619 06:58:35.791464 19870 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:35.791522 19870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:35.791679 19870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:35.793706 19870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "66d95cf6b89949f78100dc1ecf69ea20"
format_stamp: "Formatted at 2026-06-19 06:58:35 on dist-test-slave-10d0"
server_key: "ede976c1c19f09e1d0b75686e55e2398"
server_key_iv: "8f8d2844242c3e4be989c832ba4115df"
server_key_version: "encryptionkey@0"
I20260619 06:58:35.794214 19870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "66d95cf6b89949f78100dc1ecf69ea20"
format_stamp: "Formatted at 2026-06-19 06:58:35 on dist-test-slave-10d0"
server_key: "ede976c1c19f09e1d0b75686e55e2398"
server_key_iv: "8f8d2844242c3e4be989c832ba4115df"
server_key_version: "encryptionkey@0"
I20260619 06:58:35.797816 19870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:58:35.800231 19885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:35.801601 19870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:35.801743 19870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "66d95cf6b89949f78100dc1ecf69ea20"
format_stamp: "Formatted at 2026-06-19 06:58:35 on dist-test-slave-10d0"
server_key: "ede976c1c19f09e1d0b75686e55e2398"
server_key_iv: "8f8d2844242c3e4be989c832ba4115df"
server_key_version: "encryptionkey@0"
I20260619 06:58:35.801869 19870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:35.827849 19870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:35.830996 19870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:35.831208 19870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:35.838644 19870 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:36423
I20260619 06:58:35.838661 19937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:36423 every 8 connection(s)
I20260619 06:58:35.839741 19870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:35.840128 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 19870
I20260619 06:58:35.840286 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:35.840658 11689 external_mini_cluster.cc:1428] Setting key c7c35cebebb523cbfa9d7caccf7409b2
I20260619 06:58:35.843004 19938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:35.849659 19938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: Bootstrap starting.
Jun 19 06:58:35 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:35.851964 19938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:35.852903 19938 log.cc:824] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:35.854982 19938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: No bootstrap required, opened a new log
I20260619 06:58:35.855298 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:35.842033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.11.106.126:36423)
0619 06:58:35.842444 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:35.842471 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:35.842514 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:58:35.843218 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:35.843236 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:35.843566 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:35.843850 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:35.844747 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:35.845785 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:35.847002 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:35.847466 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:35.847586 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:35.850470 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:35.850475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:35.850488 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:35.850514 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:35.852315 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:35.852850 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:35.852855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:35.852859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:35.852964 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:35.853273 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:35.853275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:35.853276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:35.853804 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:35.853956 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:35.854209 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:58:35.854543 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":152,"threads_started":1}
I20260619 06:58:35.858832 19938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } }
I20260619 06:58:35.859067 19938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:35.859150 19938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d95cf6b89949f78100dc1ecf69ea20, State: Initialized, Role: FOLLOWER
I20260619 06:58:35.859570 19938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } }
I20260619 06:58:35.859719 19938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:35.859807 19938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:35.859927 19938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:35.860867 19938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } }
I20260619 06:58:35.861205 19938 leader_election.cc:302] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d95cf6b89949f78100dc1ecf69ea20; no voters:
I20260619 06:58:35.861507 19938 leader_election.cc:288] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:35.861644 19943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:35.862190 19943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 1 LEADER]: Becoming Leader. State: Replica: 66d95cf6b89949f78100dc1ecf69ea20, State: Running, Role: LEADER
I20260619 06:58:35.862697 19943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } }
I20260619 06:58:35.862836 19938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:35.864289 19945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d95cf6b89949f78100dc1ecf69ea20. Latest consensus state: current_term: 1 leader_uuid: "66d95cf6b89949f78100dc1ecf69ea20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } } }
I20260619 06:58:35.864423 19945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:35.864742 19944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66d95cf6b89949f78100dc1ecf69ea20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } } }
I20260619 06:58:35.864843 19944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:35.864825 19954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:35.868078 19954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:35.873418 19954 catalog_manager.cc:1347] Generated new cluster ID: 7c3e034e82a2474e8434970332d15297
I20260619 06:58:35.873479 19954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:35.889186 19954 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:35.890082 19954 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:35.895550 19954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: Generated new TSK 0
I20260619 06:58:35.896209 19954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:58:35.924333 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36423
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:36.042021 19964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:36.042428 19964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:36.042510 19964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:36.046701 19964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:36.046759 19964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:36.046859 19964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:36.052285 19964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:36.053385 19964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:36.054445 19964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:36.064162 19972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:36.064248 19969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:36.064162 19970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:36.064509 19964 server_base.cc:1034] running on GCE node
I20260619 06:58:36.064987 19964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:36.065588 19964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:36.066763 19964 hybrid_clock.cc:648] HybridClock initialized: now 1781852316066750 us; error 33 us; skew 500 ppm
Jun 19 06:58:36 dist-test-slave-10d0 krb5kdc[19857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:36.069484 19964 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:58:36.070339 19964 webserver.cc:460] Webserver started at http://127.11.106.65:38771/ using document root <none> and password file <none>
I20260619 06:58:36.070955 19964 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:36.071053 19964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:36.071295 19964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:36.073378 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "8e2526bf971b4e10bb755f855ffd421b"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "91d1d99534c7b8503bf10858c7d7a86a"
server_key_iv: "c3e85b16d1ed072f17de129b454f7bfb"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.073935 19964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "8e2526bf971b4e10bb755f855ffd421b"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "91d1d99534c7b8503bf10858c7d7a86a"
server_key_iv: "c3e85b16d1ed072f17de129b454f7bfb"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.077575 19964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:58:36.080076 19979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.081377 19964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:36.081525 19964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "8e2526bf971b4e10bb755f855ffd421b"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "91d1d99534c7b8503bf10858c7d7a86a"
server_key_iv: "c3e85b16d1ed072f17de129b454f7bfb"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.081625 19964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:36.098687 19964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:36.101892 19964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:36.102126 19964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:36.102775 19964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:36.103657 19964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:36.103744 19964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.103823 19964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:36.103881 19964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.113823 19964 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:35131
I20260619 06:58:36.113879 20092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:35131 every 8 connection(s)
I20260619 06:58:36.114821 19964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:36.121172 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 19964
I20260619 06:58:36.121281 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:36.121608 11689 external_mini_cluster.cc:1428] Setting key bbfbf3bf1eed927a11db2272edfd8240
Jun 19 06:58:36 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:36.129416 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:36.116712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:49975 (local address 127.11.106.126:36423)
0619 06:58:36.116862 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:36.116866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:36.117960 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 06:58:36.118913 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:36.118916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:36.118965 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:36.119039 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:36.120212 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.120828 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.122431 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.122664 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.122749 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:36.125646 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:36.125650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:36.125652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:36.125676 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:36.127259 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:36.127802 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:36.127806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:36.127807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:36.127892 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:36.128246 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:36.128251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:36.128253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:36.128451 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:36.128587 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:36.129109 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 06:58:36.129247 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:58:36.130311 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:36.117101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:49975)
0619 06:58:36.117798 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:36.117843 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:36.118669 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:36.119183 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:36.119190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:36.119557 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:36.120077 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:36.120091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.120947 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.120950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:36.122287 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:36.122298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.122849 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.122854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:36.123087 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:36.123791 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:36.123810 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:36.125466 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:36.127398 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:36.127404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:36.127421 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:36.127690 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:36.127996 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:36.127999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:36.128001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:36.128150 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:36.128592 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:36.128601 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:36.128865 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 06:58:36.129069 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 06:58:36.129440 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 06:58:36.131703 20093 heartbeater.cc:347] Connected to a master server at 127.11.106.126:36423
I20260619 06:58:36.131947 20093 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:36.132491 20093 heartbeater.cc:511] Master 127.11.106.126:36423 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 06:58:36.134320 19902 ts_manager.cc:194] Registered new tserver with Master: 8e2526bf971b4e10bb755f855ffd421b (127.11.106.65:35131)
I20260619 06:58:36.135839 19902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:49975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:58:36.152654 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36423
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:36.268673 20098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:36.268940 20098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:36.269016 20098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:36.273111 20098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:36.273173 20098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:36.273277 20098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:36.278658 20098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:36.279743 20098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:36.280846 20098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:36.290585 20106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:36.290655 20103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:36.290655 20104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:36.290813 20098 server_base.cc:1034] running on GCE node
I20260619 06:58:36.291317 20098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:36.291987 20098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:36.293172 20098 hybrid_clock.cc:648] HybridClock initialized: now 1781852316293135 us; error 49 us; skew 500 ppm
Jun 19 06:58:36 dist-test-slave-10d0 krb5kdc[19857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:36.296125 20098 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:58:36.297072 20098 webserver.cc:460] Webserver started at http://127.11.106.66:46565/ using document root <none> and password file <none>
I20260619 06:58:36.297676 20098 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:36.297771 20098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:36.298008 20098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:36.300117 20098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "e3be8af6f3134f089f425d39fc6f0113"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "7dbb2fabb8e91e9fa70e48d583ae6c26"
server_key_iv: "4dc9763578eb91117459c4fe05ce2f43"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.300772 20098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "e3be8af6f3134f089f425d39fc6f0113"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "7dbb2fabb8e91e9fa70e48d583ae6c26"
server_key_iv: "4dc9763578eb91117459c4fe05ce2f43"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.304510 20098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:36.307233 20113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.308486 20098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:36.308665 20098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "e3be8af6f3134f089f425d39fc6f0113"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "7dbb2fabb8e91e9fa70e48d583ae6c26"
server_key_iv: "4dc9763578eb91117459c4fe05ce2f43"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.308779 20098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:36.323570 20098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:36.326656 20098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:36.326864 20098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:36.327495 20098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:36.328334 20098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:36.328382 20098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.328446 20098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:36.328480 20098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.337410 20098 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:44389
I20260619 06:58:36.337428 20226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:44389 every 8 connection(s)
I20260619 06:58:36.338371 20098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:36.339319 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 20098
I20260619 06:58:36.339468 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:36.339818 11689 external_mini_cluster.cc:1428] Setting key 5791058192c334b58d2462ffa984460c
Jun 19 06:58:36 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 06:58:36.354667 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:36.340422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:60413 (local address 127.11.106.126:36423)
0619 06:58:36.340593 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:36.340598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:36.341928 (+ 1330us) server_negotiation.cc:407] Connection header received
0619 06:58:36.343327 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:36.343329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:36.343376 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:36.343450 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:36.345116 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.345653 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.347512 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.347740 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.347824 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:36.350835 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:36.350839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:36.350841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:36.350865 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:36.352243 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:36.352900 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:36.352903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:36.352904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:36.352984 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:36.353332 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:36.353335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:36.353336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:36.353628 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:36.353758 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:36.354330 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 06:58:36.354495 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:58:36.355652 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:36.340919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:60413)
0619 06:58:36.341764 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:36.341824 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:36.343076 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:36.343664 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:36.343675 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:36.344205 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:36.344968 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:36.344985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.345793 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.345800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:36.347373 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:36.347390 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.347915 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.347922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:36.348177 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:36.348970 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:36.348996 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:36.350644 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:36.352361 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:36.352366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:36.352382 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:36.352733 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:36.353065 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:36.353067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:36.353069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:36.353193 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:36.353793 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:36.353799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:36.354072 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 06:58:36.354262 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 06:58:36.354594 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":143,"threads_started":1}
I20260619 06:58:36.356945 20227 heartbeater.cc:347] Connected to a master server at 127.11.106.126:36423
I20260619 06:58:36.357221 20227 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:36.357875 20227 heartbeater.cc:511] Master 127.11.106.126:36423 requested a full tablet report, sending...
I20260619 06:58:36.359166 19902 ts_manager.cc:194] Registered new tserver with Master: e3be8af6f3134f089f425d39fc6f0113 (127.11.106.66:44389)
I20260619 06:58:36.360026 19902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:60413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:58:36.372426 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36423
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:36.486814 20232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:36.487097 20232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:36.487142 20232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:36.491295 20232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:36.491358 20232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:36.491430 20232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:36.496886 20232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:36.498054 20232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:36.499212 20232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:36.509135 20238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:36.509145 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:36.509363 20232 server_base.cc:1034] running on GCE node
W20260619 06:58:36.509447 20240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:36.509857 20232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:36.510540 20232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:36.511700 20232 hybrid_clock.cc:648] HybridClock initialized: now 1781852316511682 us; error 28 us; skew 500 ppm
Jun 19 06:58:36 dist-test-slave-10d0 krb5kdc[19857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:36.514578 20232 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:58:36.515470 20232 webserver.cc:460] Webserver started at http://127.11.106.67:45689/ using document root <none> and password file <none>
I20260619 06:58:36.516057 20232 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:36.516120 20232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:36.516347 20232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:36.518491 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "fa103759c7d3483fbe486492235e9c12"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "48b6ad6a4ac21958ead910afd2016190"
server_key_iv: "e0a608b6e705f6108cf43b6559b0fead"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.519059 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "fa103759c7d3483fbe486492235e9c12"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "48b6ad6a4ac21958ead910afd2016190"
server_key_iv: "e0a608b6e705f6108cf43b6559b0fead"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.522995 20232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:36.525614 20247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.526934 20232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:36.527079 20232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "fa103759c7d3483fbe486492235e9c12"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "48b6ad6a4ac21958ead910afd2016190"
server_key_iv: "e0a608b6e705f6108cf43b6559b0fead"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.527199 20232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:36.536284 20232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:36.539773 20232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:36.539988 20232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:36.540625 20232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:36.541508 20232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:36.541584 20232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.541657 20232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:36.541703 20232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.550580 20232 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:37559
I20260619 06:58:36.550604 20360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:37559 every 8 connection(s)
I20260619 06:58:36.551498 20232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:36.559475 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 20232
I20260619 06:58:36.559626 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:36.559988 11689 external_mini_cluster.cc:1428] Setting key 629c874060e83372c0f33a85f82b4bba
Jun 19 06:58:36 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:36.565783 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:36.553554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:42987 (local address 127.11.106.126:36423)
0619 06:58:36.553725 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:36.553728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:36.554763 (+ 1035us) server_negotiation.cc:407] Connection header received
0619 06:58:36.555649 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:36.555652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:36.555701 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:36.555776 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:36.556929 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.557462 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.558584 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.558793 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.558872 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:36.561932 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:36.561936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:36.561939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:36.561972 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:36.563548 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:36.564113 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:36.564116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:36.564117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:36.564191 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:36.564496 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:36.564499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:36.564500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:36.564882 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:36.564971 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:36.565507 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 06:58:36.565643 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:58:36.566607 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:36.553945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:42987)
0619 06:58:36.554624 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:36.554672 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:36.555463 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:36.555942 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:36.555948 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:36.556317 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:36.556809 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:36.556822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.557573 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.557576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:36.558475 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:36.558484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:36.558923 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:36.558926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:36.559061 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:36.559796 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:36.559822 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:36.561761 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:36.563716 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:36.563725 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:36.563744 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:36.564028 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:36.564262 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:36.564267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:36.564274 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:36.564410 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:36.564990 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:36.564996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:36.565257 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 06:58:36.565423 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 06:58:36.565687 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":117,"threads_started":1}
I20260619 06:58:36.567859 20361 heartbeater.cc:347] Connected to a master server at 127.11.106.126:36423
I20260619 06:58:36.568135 20361 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:36.568713 20361 heartbeater.cc:511] Master 127.11.106.126:36423 requested a full tablet report, sending...
I20260619 06:58:36.569799 19902 ts_manager.cc:194] Registered new tserver with Master: fa103759c7d3483fbe486492235e9c12 (127.11.106.67:37559)
I20260619 06:58:36.570369 19902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:42987
I20260619 06:58:36.574311 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab.
Entry for principal oryx/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab.
WARNING: no policy specified for oryx/127.11.106.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab.
Entry for principal oryx/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab.
WARNING: no policy specified for oryx/127.11.106.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab.
Entry for principal oryx/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab.
I20260619 06:58:36.651906 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 19964
I20260619 06:58:36.657778 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 20098
I20260619 06:58:36.663568 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 20232
I20260619 06:58:36.668916 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 19870
I20260619 06:58:36.675422 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36423
--webserver_interface=127.11.106.126
--webserver_port=43817
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:36423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:36.787617 20370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:36.787948 20370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:36.788034 20370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:36.792081 20370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:36.792188 20370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:36.792228 20370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:36.792287 20370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:36.792326 20370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:36.798096 20370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:36423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=43817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:36.799256 20370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:36.800343 20370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:36.809338 20375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:36.809360 20376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:36.809453 20370 server_base.cc:1034] running on GCE node
W20260619 06:58:36.809338 20378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:36.810160 20370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:36.811396 20370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:36.812655 20370 hybrid_clock.cc:648] HybridClock initialized: now 1781852316812637 us; error 28 us; skew 500 ppm
Jun 19 06:58:36 dist-test-slave-10d0 krb5kdc[19857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852316, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:36.815505 20370 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:36.816425 20370 webserver.cc:460] Webserver started at http://127.11.106.126:43817/ using document root <none> and password file <none>
I20260619 06:58:36.817055 20370 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:36.817169 20370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:36.821142 20370 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:58:36.823381 20385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:36.824664 20370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:36.824839 20370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "66d95cf6b89949f78100dc1ecf69ea20"
format_stamp: "Formatted at 2026-06-19 06:58:35 on dist-test-slave-10d0"
server_key: "ede976c1c19f09e1d0b75686e55e2398"
server_key_iv: "8f8d2844242c3e4be989c832ba4115df"
server_key_version: "encryptionkey@0"
I20260619 06:58:36.825397 20370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:36.842427 20370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:36.845382 20370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:36.845590 20370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:36.852771 20370 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:36423
I20260619 06:58:36.852794 20437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:36423 every 8 connection(s)
I20260619 06:58:36.853941 20370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:36.858163 20438 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:58:36.859824 20438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: Bootstrap starting.
I20260619 06:58:36.862076 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 20370
I20260619 06:58:36.862525 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:35131
--local_ip_for_outbound_sockets=127.11.106.65
--tserver_master_addrs=127.11.106.126:36423
--webserver_port=38771
--webserver_interface=127.11.106.65
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.106.65
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:58:36.877086 20438 log.cc:824] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:36.883903 20438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:58:36.884292 20438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: Bootstrap complete.
I20260619 06:58:36.888734 20438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } }
I20260619 06:58:36.889163 20438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d95cf6b89949f78100dc1ecf69ea20, State: Initialized, Role: FOLLOWER
I20260619 06:58:36.889703 20438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } }
I20260619 06:58:36.889869 20438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:36.889953 20438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:36.890072 20438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:58:36.892022 20438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } }
I20260619 06:58:36.892369 20438 leader_election.cc:302] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d95cf6b89949f78100dc1ecf69ea20; no voters:
I20260619 06:58:36.892760 20438 leader_election.cc:288] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:58:36.892917 20442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:58:36.893170 20442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [term 2 LEADER]: Becoming Leader. State: Replica: 66d95cf6b89949f78100dc1ecf69ea20, State: Running, Role: LEADER
I20260619 06:58:36.893494 20442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } }
I20260619 06:58:36.894049 20438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:36.894502 20442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d95cf6b89949f78100dc1ecf69ea20. Latest consensus state: current_term: 2 leader_uuid: "66d95cf6b89949f78100dc1ecf69ea20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } } }
I20260619 06:58:36.894621 20442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:36.895231 20443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "66d95cf6b89949f78100dc1ecf69ea20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d95cf6b89949f78100dc1ecf69ea20" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36423 } } }
I20260619 06:58:36.895334 20443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:36.896315 20446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:36.899919 20446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:36.900520 20446 catalog_manager.cc:1259] Loaded cluster ID: 7c3e034e82a2474e8434970332d15297
I20260619 06:58:36.900599 20446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:36.902930 20446 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:36.903599 20446 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66d95cf6b89949f78100dc1ecf69ea20: Loaded TSK: 0
I20260619 06:58:36.904816 20446 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:58:36.978271 20440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:36.978564 20440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:36.978610 20440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:36.982908 20440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:36.982981 20440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:36.983072 20440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:36.988691 20440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.65.keytab
--principal=oryx/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:35131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=38771
--tserver_master_addrs=127.11.106.126:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:36.989805 20440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:36.990854 20440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:37.000705 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:37.000847 20467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:37.000728 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:37.000991 20440 server_base.cc:1034] running on GCE node
I20260619 06:58:37.001454 20440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:37.002154 20440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:37.003338 20440 hybrid_clock.cc:648] HybridClock initialized: now 1781852317003306 us; error 48 us; skew 500 ppm
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852317, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:37.006126 20440 init.cc:377] Logged in from keytab as oryx/127.11.106.65@KRBTEST.COM (short username oryx)
I20260619 06:58:37.007092 20440 webserver.cc:460] Webserver started at http://127.11.106.65:38771/ using document root <none> and password file <none>
I20260619 06:58:37.007663 20440 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:37.007745 20440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:37.011765 20440 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:37.013924 20474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:37.015204 20440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:58:37.015295 20440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "8e2526bf971b4e10bb755f855ffd421b"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "91d1d99534c7b8503bf10858c7d7a86a"
server_key_iv: "c3e85b16d1ed072f17de129b454f7bfb"
server_key_version: "encryptionkey@0"
I20260619 06:58:37.015846 20440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:37.032311 20440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:37.035645 20440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:37.035861 20440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:37.036523 20440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:37.037433 20440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:37.037482 20440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:37.037523 20440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:37.037575 20440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:37.046413 20440 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:35131
I20260619 06:58:37.046442 20587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:35131 every 8 connection(s)
I20260619 06:58:37.047398 20440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:37.049244 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 20440
I20260619 06:58:37.049748 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:44389
--local_ip_for_outbound_sockets=127.11.106.66
--tserver_master_addrs=127.11.106.126:36423
--webserver_port=46565
--webserver_interface=127.11.106.66
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.106.66
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:58:37.062857 20590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:37.049603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:60285 (local address 127.11.106.126:36423)
0619 06:58:37.050373 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.050394 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.050754 (+ 360us) server_negotiation.cc:407] Connection header received
0619 06:58:37.052076 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.052096 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.052521 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.052888 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.054429 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.055401 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.057309 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.057794 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.057914 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.062143 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:60285: BlockingRecv error: recv got EOF from 127.11.106.65:60285 (error 108)
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":127,"threads_started":1}
I20260619 06:58:37.062901 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.049666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:60285)
0619 06:58:37.050564 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:37.050624 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:37.051583 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:37.053013 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:37.053025 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:37.053538 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:37.054245 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.054271 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.055554 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.055560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.057181 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.057196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.057973 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.057981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.058203 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.058943 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:37.058968 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:37.061941 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":163,"threads_started":1}
W20260619 06:58:37.063373 20588 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:36423 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.106.126:36423: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:37.069347 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.063801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:35339)
0619 06:58:37.063943 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:37.063956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:37.064032 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:37.064351 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:37.064354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:37.064518 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:37.064645 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.064651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.065418 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.065421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.066306 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.066316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.066732 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.066735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.066828 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.067247 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:37.067261 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:37.069212 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 06:58:37.069455 20590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:37.063911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:35339 (local address 127.11.106.126:36423)
0619 06:58:37.064020 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.064023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.064035 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:58:37.064136 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.064138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.064226 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.064312 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.064747 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.065320 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.066419 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.066630 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.066704 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.069336 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:35339: BlockingRecv error: recv got EOF from 127.11.106.65:35339 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:37.074973 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.069964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:39769)
0619 06:58:37.070073 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:37.070090 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:37.070164 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:37.070372 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:37.070375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:37.070495 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:37.070564 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.070568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.071556 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.071558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.072070 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.072077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.072649 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.072651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.072709 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.073176 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:37.073186 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:37.074875 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:58:37.075140 20590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:37.070040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:39769 (local address 127.11.106.126:36423)
0619 06:58:37.070139 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.070143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.070156 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:58:37.070227 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.070230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.070270 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.070350 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.070662 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.071441 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.072172 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.072518 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.072644 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.075030 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:39769: BlockingRecv error: recv got EOF from 127.11.106.65:39769 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 06:58:37.075371 20588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:58:37.166721 20592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:37.166991 20592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:37.167071 20592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:37.171157 20592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:37.171222 20592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:37.171283 20592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:37.176625 20592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.66.keytab
--principal=oryx/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:44389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=46565
--tserver_master_addrs=127.11.106.126:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:37.177702 20592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:37.178747 20592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:37.188647 20597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:37.188616 20598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:37.188612 20600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:37.188932 20592 server_base.cc:1034] running on GCE node
I20260619 06:58:37.189298 20592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:37.189919 20592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:37.191077 20592 hybrid_clock.cc:648] HybridClock initialized: now 1781852317191060 us; error 31 us; skew 500 ppm
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852317, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:37.193832 20592 init.cc:377] Logged in from keytab as oryx/127.11.106.66@KRBTEST.COM (short username oryx)
I20260619 06:58:37.194749 20592 webserver.cc:460] Webserver started at http://127.11.106.66:46565/ using document root <none> and password file <none>
I20260619 06:58:37.195299 20592 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:37.195407 20592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:37.199316 20592 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:58:37.201478 20607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:37.202803 20592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:58:37.202960 20592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "e3be8af6f3134f089f425d39fc6f0113"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "7dbb2fabb8e91e9fa70e48d583ae6c26"
server_key_iv: "4dc9763578eb91117459c4fe05ce2f43"
server_key_version: "encryptionkey@0"
I20260619 06:58:37.203490 20592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:37.214191 20592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:37.217334 20592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:37.217527 20592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:37.218104 20592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:37.219065 20592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:37.219133 20592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:37.219200 20592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:37.219244 20592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:37.228330 20592 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:44389
I20260619 06:58:37.228355 20720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:44389 every 8 connection(s)
I20260619 06:58:37.229471 20592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:37.237176 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 20592
I20260619 06:58:37.237632 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:37559
--local_ip_for_outbound_sockets=127.11.106.67
--tserver_master_addrs=127.11.106.126:36423
--webserver_port=45689
--webserver_interface=127.11.106.67
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.106.67
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:58:37.240345 20590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:37.231293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:59047 (local address 127.11.106.126:36423)
0619 06:58:37.231437 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.231441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.232490 (+ 1049us) server_negotiation.cc:407] Connection header received
0619 06:58:37.233443 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.233446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.233510 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.233613 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.234777 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.235305 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.236445 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.236744 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.236859 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.240219 (+ 3360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:59047: BlockingRecv error: recv got EOF from 127.11.106.66:59047 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:58:37.241015 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.231687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:59047)
0619 06:58:37.232347 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:37.232387 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:37.233226 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:37.233801 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:37.233809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:37.234171 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:37.234632 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.234646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.235429 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.235433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.236317 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.236326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.236898 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.236902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.237071 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.237734 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:37.237761 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:37.240098 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":111,"threads_started":1}
W20260619 06:58:37.241580 20721 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:36423 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.106.126:36423: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:37.248229 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.242068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:58313)
0619 06:58:37.242277 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:37.242294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:37.242396 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:37.242652 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:37.242658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:37.242863 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:37.243018 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.243029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.243780 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.243787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.244870 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.244884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.245347 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.245351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.245479 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.246008 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:37.246029 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:37.248059 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124}
W20260619 06:58:37.248291 20590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:37.242160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:58313 (local address 127.11.106.126:36423)
0619 06:58:37.242275 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.242279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.242342 (+ 63us) server_negotiation.cc:407] Connection header received
0619 06:58:37.242490 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.242492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.242531 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.242599 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.243150 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.243632 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.245010 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.245222 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.245303 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.248182 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:58313: BlockingRecv error: recv got EOF from 127.11.106.66:58313 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:37.254019 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.249126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:39287)
0619 06:58:37.249247 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:37.249261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:37.249338 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:37.249566 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:37.249569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:37.249720 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:37.249786 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.249790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.250495 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.250498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.251015 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.251023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.251495 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.251499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.251583 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.252063 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:37.252075 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:37.253859 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 06:58:37.254082 20590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:37.249208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:39287 (local address 127.11.106.126:36423)
0619 06:58:37.249334 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.249338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.249352 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:58:37.249420 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.249423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.249470 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.249552 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.249896 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.250391 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.251130 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.251377 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.251476 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.253973 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:39287: BlockingRecv error: recv got EOF from 127.11.106.66:39287 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 06:58:37.254266 20721 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:58:37.350602 20724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:37.350859 20724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:37.350901 20724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:37.354920 20724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:37.354974 20724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:37.355031 20724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:37.360190 20724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/krb5kdc/oryx_127.11.106.67.keytab
--principal=oryx/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:37559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=45689
--tserver_master_addrs=127.11.106.126:36423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:37.361249 20724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:37.362206 20724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:37.371968 20730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:37.371987 20729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:37.371976 20732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:37.372326 20724 server_base.cc:1034] running on GCE node
I20260619 06:58:37.372794 20724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:37.373499 20724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:37.374687 20724 hybrid_clock.cc:648] HybridClock initialized: now 1781852317374668 us; error 32 us; skew 500 ppm
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852317, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:37.377488 20724 init.cc:377] Logged in from keytab as oryx/127.11.106.67@KRBTEST.COM (short username oryx)
I20260619 06:58:37.378298 20724 webserver.cc:460] Webserver started at http://127.11.106.67:45689/ using document root <none> and password file <none>
I20260619 06:58:37.378837 20724 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:37.378908 20724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:37.382678 20724 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 06:58:37.384696 20739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:37.385895 20724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:58:37.386021 20724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "fa103759c7d3483fbe486492235e9c12"
format_stamp: "Formatted at 2026-06-19 06:58:36 on dist-test-slave-10d0"
server_key: "48b6ad6a4ac21958ead910afd2016190"
server_key_iv: "e0a608b6e705f6108cf43b6559b0fead"
server_key_version: "encryptionkey@0"
I20260619 06:58:37.386530 20724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:37.428690 20724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:37.432384 20724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:37.432612 20724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:37.433270 20724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:37.434113 20724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:37.434158 20724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:37.434196 20724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:37.434212 20724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:37.443077 20724 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:37559
I20260619 06:58:37.443105 20852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:37559 every 8 connection(s)
I20260619 06:58:37.444203 20724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:37.444597 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 20724
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:37.455614 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.445461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.11.106.126:36423)
0619 06:58:37.445613 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.445619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.445837 (+ 218us) server_negotiation.cc:407] Connection header received
0619 06:58:37.446029 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.446032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.446082 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.446174 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.447068 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.447912 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.448934 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.449269 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.449379 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.451022 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:37.451029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:37.451042 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:37.451071 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:37.453148 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:37.453624 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:37.453631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:37.453637 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:37.453833 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:37.454147 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:37.454151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:37.454152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:37.454629 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:37.454870 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:37.455160 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 06:58:37.455424 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:58:37.457230 20856 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:37.446661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:57327 (local address 127.11.106.126:36423)
0619 06:58:37.446938 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.446942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.447857 (+ 915us) server_negotiation.cc:407] Connection header received
0619 06:58:37.448919 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.448923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.448975 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.449056 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.450399 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.451168 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.452939 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.453165 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.453288 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.457130 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:57327: BlockingRecv error: recv got EOF from 127.11.106.67:57327 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
I20260619 06:58:37.458042 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.446854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:57327)
0619 06:58:37.447680 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:37.447738 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:37.448692 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:37.449274 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:37.449284 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:37.449721 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:37.450230 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.450243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.451313 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.451318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.452806 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.452818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.453351 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.453354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.453500 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.454078 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:37.454113 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:37.457026 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":124,"threads_started":1}
W20260619 06:58:37.458583 20853 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:36423 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.106.126:36423: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:37.463896 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.459035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:33563)
0619 06:58:37.459154 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:37.459171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:37.459246 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:37.459484 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:37.459487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:37.459645 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:37.459741 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.459746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.460488 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.460494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.461102 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.461111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.461585 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.461588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.461681 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.462097 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:37.462107 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:37.463746 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:58:37.463984 20856 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:37.459123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:33563 (local address 127.11.106.126:36423)
0619 06:58:37.459240 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.459244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.459270 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:58:37.459333 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.459336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.459376 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.459468 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.459862 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.460378 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.461227 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.461470 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.461581 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.463866 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:33563: BlockingRecv error: recv got EOF from 127.11.106.67:33563 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:37 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:37.469182 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:37.464500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:41043)
0619 06:58:37.464681 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:37.464700 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:37.464781 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:37.464969 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:37.464971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:37.465090 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:37.465156 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.465160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.465928 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.465930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.466445 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:37.466452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.466869 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.466872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:37.466927 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.467296 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:37.467306 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:37.469026 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260619 06:58:37.469241 20856 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:37.464609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:41043 (local address 127.11.106.126:36423)
0619 06:58:37.464794 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:37.464797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:37.464812 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:37.464841 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:37.464843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:37.464876 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:37.464941 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:37.465248 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.465811 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.466539 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:37.466776 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:37.466851 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:37.469137 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:41043: BlockingRecv error: recv got EOF from 127.11.106.67:41043 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260619 06:58:37.469439 20853 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 06:58:38 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:38 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:38.081401 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:38.075990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:51111)
0619 06:58:38.076264 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:38.076278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:38.076366 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:38.076604 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:38.076606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:38.076793 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:38.076916 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:38.076921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.077660 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.077663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:38.078218 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:38.078225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.078688 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.078692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:38.078774 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:38.079381 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:38.079393 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:38.081274 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
W20260619 06:58:38.081620 20859 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:38.076082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:51111 (local address 127.11.106.126:36423)
0619 06:58:38.076355 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:38.076358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:38.076371 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:58:38.076449 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:38.076451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:38.076494 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:38.076587 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:38.077023 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.077571 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.078326 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.078556 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.078633 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:38.081430 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:51111: BlockingRecv error: recv got EOF from 127.11.106.65:51111 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
Jun 19 06:58:38 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:38 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:38.260592 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:38.254922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:54963)
0619 06:58:38.255224 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:38.255239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:38.255353 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:38.255607 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:38.255609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:38.255806 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:38.255932 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:38.255938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.256694 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.256697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:38.257277 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:38.257286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.257829 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.257833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:38.257963 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:38.258572 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:38.258594 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:38.260426 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":118,"threads_started":1}
W20260619 06:58:38.260735 20859 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:38.255027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:54963 (local address 127.11.106.126:36423)
0619 06:58:38.255177 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:38.255181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:38.255312 (+ 131us) server_negotiation.cc:407] Connection header received
0619 06:58:38.255442 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:38.255444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:38.255490 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:38.255564 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:38.256037 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.256561 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.257404 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.257691 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.257816 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:38.260591 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:54963: BlockingRecv error: recv got EOF from 127.11.106.66:54963 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 06:58:38 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:38 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:38.475543 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:38.470114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:58257)
0619 06:58:38.470409 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:38.470421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:38.470519 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:38.470748 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:38.470751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:38.470922 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:38.471059 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:38.471064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.471830 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.471833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:38.472383 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:38.472390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.472870 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.472873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:38.472951 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:38.473554 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:38.473575 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:38.475417 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
W20260619 06:58:38.475652 20859 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:38.470199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:58257 (local address 127.11.106.126:36423)
0619 06:58:38.470353 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:38.470357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:38.470488 (+ 131us) server_negotiation.cc:407] Connection header received
0619 06:58:38.470597 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:38.470599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:38.470645 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:38.470711 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:38.471166 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.471734 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.472487 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:38.472746 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:38.472827 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:38.475550 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:58257: BlockingRecv error: recv got EOF from 127.11.106.67:58257 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:58:39 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:39 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:39.091113 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:39.082368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:53411)
0619 06:58:39.082751 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:39.082767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:39.082892 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:39.083209 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:39.083212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:39.083461 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:39.083604 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:39.083624 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.084386 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.084389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:39.085022 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:39.085031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.085467 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.085470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:39.085536 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:39.089058 (+ 3522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:39.089069 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:39.090982 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":82,"threads_started":1}
W20260619 06:58:39.091264 20863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:39.082431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:53411 (local address 127.11.106.126:36423)
0619 06:58:39.082766 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:39.082770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:39.082848 (+ 78us) server_negotiation.cc:407] Connection header received
0619 06:58:39.083000 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:39.083003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:39.083067 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:39.083166 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:39.083758 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.084291 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.085129 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.085355 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.085435 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:39.091126 (+ 5691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:53411: BlockingRecv error: recv got EOF from 127.11.106.65:53411 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":92,"threads_started":1}
Jun 19 06:58:39 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:39 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:39.267170 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:39.261460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:49627)
0619 06:58:39.261745 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:39.261758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:39.261856 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:39.262076 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:39.262079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:39.262251 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:39.262346 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:39.262351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.263165 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.263177 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:39.263802 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:39.263813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.264311 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.264314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:39.264394 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:39.264998 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:39.265010 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:39.267012 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
W20260619 06:58:39.267324 20863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:39.261489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:49627 (local address 127.11.106.126:36423)
0619 06:58:39.261662 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:39.261667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:39.261871 (+ 204us) server_negotiation.cc:407] Connection header received
0619 06:58:39.261924 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:39.261926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:39.261982 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:39.262059 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:39.262450 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.263040 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.263932 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.264169 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.264272 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:39.267158 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:49627: BlockingRecv error: recv got EOF from 127.11.106.66:49627 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 06:58:39 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:39 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:39.482049 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:39.476479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:41141)
0619 06:58:39.476806 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:39.476824 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:39.476932 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:39.477182 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:39.477184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:39.477359 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:39.477456 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:39.477461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.478264 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.478267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:39.478855 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:39.478865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.479311 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.479314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:39.479376 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:39.479993 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:39.480006 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:39.481919 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":177,"threads_started":1}
W20260619 06:58:39.482163 20863 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:39.476631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:41141 (local address 127.11.106.126:36423)
0619 06:58:39.476794 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:39.476798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:39.476903 (+ 105us) server_negotiation.cc:407] Connection header received
0619 06:58:39.477018 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:39.477020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:39.477067 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:39.477144 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:39.477569 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.478139 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.478988 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:39.479202 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:39.479292 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:39.482048 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:41141: BlockingRecv error: recv got EOF from 127.11.106.67:41141 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 06:58:40 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:40 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:40.103917 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:40.091960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:40257)
0619 06:58:40.092258 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:40.092273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:40.092370 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:40.092688 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:40.092691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:40.092890 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:40.093017 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:40.093023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.093805 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.093809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:40.094550 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:40.094562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.094995 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.094998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:40.095065 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:40.101762 (+ 6697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:40.101776 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:40.103777 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
W20260619 06:58:40.104048 20867 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:40.092074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:40257 (local address 127.11.106.126:36423)
0619 06:58:40.092333 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:40.092337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:40.092350 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:58:40.092459 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:40.092461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:40.092505 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:40.092606 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:40.093134 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.093688 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.094656 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.094890 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.094971 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:40.103915 (+ 8944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:40257: BlockingRecv error: recv got EOF from 127.11.106.65:40257 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":113,"threads_started":1}
Jun 19 06:58:40 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:40 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:40.274214 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:40.268068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:41527)
0619 06:58:40.268389 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:40.268404 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:40.268510 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:40.268826 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:40.268829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:40.269035 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:40.269141 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:40.269147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.269945 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.269948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:40.270586 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:40.270596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.271180 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.271184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:40.271254 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:40.271817 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:40.271829 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:40.274059 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":126,"threads_started":1}
W20260619 06:58:40.274412 20867 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:40.268182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:41527 (local address 127.11.106.126:36423)
0619 06:58:40.268331 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:40.268335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:40.268458 (+ 123us) server_negotiation.cc:407] Connection header received
0619 06:58:40.268626 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:40.268629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:40.268710 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:40.268776 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:40.269255 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.269820 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.270767 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.271031 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.271147 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:40.274232 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:41527: BlockingRecv error: recv got EOF from 127.11.106.66:41527 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 06:58:40 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:40 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:40.488783 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:40.482954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:33425)
0619 06:58:40.483251 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:40.483265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:40.483369 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:40.483624 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:40.483626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:40.483806 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:40.483905 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:40.483910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.484665 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.484668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:40.485237 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:40.485245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.485664 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.485666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:40.485727 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:40.486214 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:40.486226 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:40.488588 (+ 2362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
W20260619 06:58:40.488878 20867 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:40.483129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:33425 (local address 127.11.106.126:36423)
0619 06:58:40.483275 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:40.483280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:40.483317 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:58:40.483455 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:40.483457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:40.483503 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:40.483572 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:40.484007 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.484531 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.485337 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:40.485539 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:40.485612 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:40.488759 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:33425: BlockingRecv error: recv got EOF from 127.11.106.67:33425 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 06:58:41 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:41 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:41.110360 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:41.104834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:48121)
0619 06:58:41.105105 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:41.105119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:41.105226 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:41.105500 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:41.105503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:41.105678 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:41.105784 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:41.105789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.106529 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.106533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:41.107091 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:41.107100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.107684 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.107689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:41.107778 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:41.108267 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:41.108279 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:41.110193 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
W20260619 06:58:41.110581 20871 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:41.104987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:48121 (local address 127.11.106.126:36423)
0619 06:58:41.105202 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:41.105205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:41.105219 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:58:41.105343 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:41.105346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:41.105392 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:41.105463 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:41.105891 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.106426 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.107278 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.107512 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.107630 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:41.110346 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:48121: BlockingRecv error: recv got EOF from 127.11.106.65:48121 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
Jun 19 06:58:41 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:41 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:41.281540 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:41.275179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:51335)
0619 06:58:41.275482 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:41.275496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:41.275599 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:41.275880 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:41.275883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:41.276072 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:41.276175 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:41.276180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.277090 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.277093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:41.277705 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:41.277716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.278226 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.278230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:41.278304 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:41.279028 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:41.279045 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:41.281413 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":123,"threads_started":1}
W20260619 06:58:41.281716 20871 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:41.275277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:51335 (local address 127.11.106.126:36423)
0619 06:58:41.275424 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:41.275428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:41.275565 (+ 137us) server_negotiation.cc:407] Connection header received
0619 06:58:41.275682 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:41.275684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:41.275761 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:41.275840 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:41.276313 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.276950 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.277851 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.278078 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.278166 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:41.281549 (+ 3383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:51335: BlockingRecv error: recv got EOF from 127.11.106.66:51335 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:58:41 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:41 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:41.495831 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:41.489586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:49093)
0619 06:58:41.490008 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:41.490020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:41.490162 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:41.490448 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:41.490452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:41.490703 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:41.490798 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:41.490804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.491638 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.491643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:41.492211 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:41.492219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.492765 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.492769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:41.492854 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:41.493451 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:41.493469 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:41.495707 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":133,"threads_started":1}
W20260619 06:58:41.496064 20871 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:41.489718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:49093 (local address 127.11.106.126:36423)
0619 06:58:41.489958 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:41.489963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:41.490097 (+ 134us) server_negotiation.cc:407] Connection header received
0619 06:58:41.490280 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:41.490288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:41.490337 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:41.490425 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:41.490926 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.491512 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.492350 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:41.492648 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:41.492760 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:41.495832 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:49093: BlockingRecv error: recv got EOF from 127.11.106.67:49093 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 06:58:42 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:42 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:42.117723 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:42.111311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:49527)
0619 06:58:42.111625 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:42.111639 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:42.111755 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:42.112029 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:42.112031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:42.112248 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:42.112354 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:42.112360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.113196 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.113199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:42.114107 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:42.114119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.114832 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.114840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:42.114927 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:42.115542 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:42.115565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:42.117564 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":131,"threads_started":1}
W20260619 06:58:42.118018 20875 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:42.111408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:49527 (local address 127.11.106.126:36423)
0619 06:58:42.111711 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:42.111715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:42.111730 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:42.111853 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:42.111855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:42.111902 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:42.111985 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:42.112465 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.113062 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.114269 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.114638 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.114784 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:42.117767 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:49527: BlockingRecv error: recv got EOF from 127.11.106.65:49527 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
Jun 19 06:58:42 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:42 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:42.288022 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:42.282382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:34441)
0619 06:58:42.282707 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:42.282721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:42.282814 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:42.283055 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:42.283057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:42.283254 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:42.283353 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:42.283358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.284105 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.284108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:42.284819 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:42.284829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.285239 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.285242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:42.285304 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:42.285905 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:42.285918 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:42.287847 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":136,"threads_started":1}
W20260619 06:58:42.288082 20875 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:42.282531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:34441 (local address 127.11.106.126:36423)
0619 06:58:42.282688 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:42.282692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:42.282791 (+ 99us) server_negotiation.cc:407] Connection header received
0619 06:58:42.282898 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:42.282900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:42.282942 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:42.283010 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:42.283454 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.284001 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.284925 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.285143 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.285217 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:42.287976 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:34441: BlockingRecv error: recv got EOF from 127.11.106.66:34441 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:58:42 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:42 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:42.502835 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:42.496724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:49051)
0619 06:58:42.496989 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:42.497003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:42.497098 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:42.497360 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:42.497362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:42.497545 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:42.497675 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:42.497681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.498406 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.498409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:42.499000 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:42.499009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.499451 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.499454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:42.499525 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:42.500092 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:42.500104 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:42.502668 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
W20260619 06:58:42.502871 20875 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:42.496789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:49051 (local address 127.11.106.126:36423)
0619 06:58:42.496935 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:42.496939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:42.497075 (+ 136us) server_negotiation.cc:407] Connection header received
0619 06:58:42.497184 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:42.497186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:42.497233 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:42.497321 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:42.497782 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.498300 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.499105 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:42.499328 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:42.499405 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:42.502768 (+ 3363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:49051: BlockingRecv error: recv got EOF from 127.11.106.67:49051 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 06:58:43 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:43 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:43.124166 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:43.118604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:58187)
0619 06:58:43.118888 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:43.118905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:43.119021 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:43.119274 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:43.119276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:43.119454 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:43.119549 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:43.119554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.120508 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.120514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:43.121128 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:43.121137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.121617 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.121620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:43.121709 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:43.122204 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:43.122215 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:43.124050 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
W20260619 06:58:43.124426 20879 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:43.118774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:58187 (local address 127.11.106.126:36423)
0619 06:58:43.119075 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:43.119079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:43.119093 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:58:43.119133 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:43.119135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:43.119177 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:43.119258 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:43.119716 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.120386 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.121268 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.121499 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.121612 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:43.124266 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:58187: BlockingRecv error: recv got EOF from 127.11.106.65:58187 (error 108)
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":84,"threads_started":1}
Jun 19 06:58:43 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:43 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:43.294962 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:43.288893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:34091)
0619 06:58:43.289229 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:43.289243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:43.289356 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:43.289664 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:43.289667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:43.289877 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:43.290072 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:43.290081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.290884 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.290887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:43.291462 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:43.291471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.291980 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.291983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:43.292051 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:43.292644 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:43.292656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:43.294779 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":151,"threads_started":1}
W20260619 06:58:43.295032 20879 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:43.288974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:34091 (local address 127.11.106.126:36423)
0619 06:58:43.289169 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:43.289174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:43.289292 (+ 118us) server_negotiation.cc:407] Connection header received
0619 06:58:43.289422 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:43.289424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:43.289468 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:43.289535 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:43.290216 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.290769 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.291579 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.291841 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.291926 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:43.294922 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:34091: BlockingRecv error: recv got EOF from 127.11.106.66:34091 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 06:58:43 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:43 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:43.509778 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:43.503691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:40421)
0619 06:58:43.504013 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:43.504027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:43.504138 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:43.504393 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:43.504396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:43.504617 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:43.504725 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:43.504731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.505714 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.505721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:43.506644 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:43.506655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.507174 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.507177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:43.507249 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:43.507844 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:43.507857 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:43.509658 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":124,"threads_started":1}
W20260619 06:58:43.509943 20879 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:43.503763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:40421 (local address 127.11.106.126:36423)
0619 06:58:43.503905 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:43.503909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:43.504102 (+ 193us) server_negotiation.cc:407] Connection header received
0619 06:58:43.504227 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:43.504229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:43.504275 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:43.504346 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:43.504860 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.505569 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.506759 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:43.506997 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:43.507077 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:43.509809 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:40421: BlockingRecv error: recv got EOF from 127.11.106.67:40421 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 06:58:44 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:44 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:44.130961 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:44.124982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:46747)
0619 06:58:44.125327 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:44.125340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:44.125434 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:44.125741 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:44.125743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:44.125925 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:44.126018 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:44.126023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.126903 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.126906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:44.127513 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:44.127522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.128010 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.128014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:44.128100 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:44.128733 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:44.128744 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:44.130786 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":131,"threads_started":1}
W20260619 06:58:44.131045 20883 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:44.125157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:46747 (local address 127.11.106.126:36423)
0619 06:58:44.125406 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:44.125411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:44.125426 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:44.125522 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:44.125525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:44.125572 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:44.125703 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:44.126148 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.126786 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.127644 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.127889 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.128006 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:44.130918 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:46747: BlockingRecv error: recv got EOF from 127.11.106.65:46747 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
Jun 19 06:58:44 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:44 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:44.301563 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:44.295962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:53319)
0619 06:58:44.296296 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:44.296309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:44.296404 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:44.296775 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:44.296778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:44.296962 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:44.297058 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:44.297063 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.297860 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.297863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:44.298426 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:44.298434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.298896 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.298899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:44.298957 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:44.299458 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:44.299469 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:44.301448 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":97,"threads_started":1}
W20260619 06:58:44.301803 20883 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:44.296022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:53319 (local address 127.11.106.126:36423)
0619 06:58:44.296173 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:44.296177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:44.296363 (+ 186us) server_negotiation.cc:407] Connection header received
0619 06:58:44.296504 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:44.296508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:44.296624 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:44.296710 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:44.297182 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.297755 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.298536 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.298794 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.298884 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:44.301672 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:53319: BlockingRecv error: recv got EOF from 127.11.106.66:53319 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 06:58:44 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:44 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:44.516296 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:44.510532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:49349)
0619 06:58:44.510870 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:44.510884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:44.510986 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:44.511255 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:44.511259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:44.511446 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:44.511539 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:44.511545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.512368 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.512371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:44.513070 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:44.513084 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.513537 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.513540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:44.513608 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:44.514181 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:44.514193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:44.516175 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":91,"threads_started":1}
W20260619 06:58:44.516633 20883 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:44.510642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:49349 (local address 127.11.106.126:36423)
0619 06:58:44.510845 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:44.510849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:44.510954 (+ 105us) server_negotiation.cc:407] Connection header received
0619 06:58:44.511066 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:44.511068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:44.511116 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:44.511188 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:44.511714 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.512254 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.513189 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:44.513406 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:44.513485 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:44.516460 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:49349: BlockingRecv error: recv got EOF from 127.11.106.67:49349 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 06:58:45 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:45 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:45.143369 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:45.131916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:56533)
0619 06:58:45.132238 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:45.132269 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:45.132414 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:45.132778 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:45.132782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:45.133031 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:45.133183 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:45.133217 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.134327 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.134331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:45.134983 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:45.134994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.135464 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.135467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:45.135539 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:45.140932 (+ 5393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:45.140949 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:45.143218 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
W20260619 06:58:45.143580 20887 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:45.132091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:56533 (local address 127.11.106.126:36423)
0619 06:58:45.132374 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:45.132393 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:45.132407 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:58:45.132525 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:45.132527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:45.132626 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:45.132704 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:45.133364 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.134196 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.135113 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.135349 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.135431 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:45.143417 (+ 7986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:56533: BlockingRecv error: recv got EOF from 127.11.106.65:56533 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":112,"threads_started":1}
Jun 19 06:58:45 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:45 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:45.308264 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:45.302375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:48933)
0619 06:58:45.302697 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:45.302715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:45.302830 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:45.303078 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:45.303080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:45.303282 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:45.303386 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:45.303392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.304290 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.304293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:45.304907 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:45.304919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.305373 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.305376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:45.305443 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:45.306059 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:45.306072 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:45.308125 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":141,"threads_started":1}
W20260619 06:58:45.308509 20887 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:45.302421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:48933 (local address 127.11.106.126:36423)
0619 06:58:45.302558 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:45.302562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:45.302792 (+ 230us) server_negotiation.cc:407] Connection header received
0619 06:58:45.302916 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:45.302918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:45.302962 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:45.303029 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:45.303495 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.304171 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.305019 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.305267 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.305348 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:45.308275 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:48933: BlockingRecv error: recv got EOF from 127.11.106.66:48933 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 06:58:45 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:45 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:45.523314 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:45.517145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:51419)
0619 06:58:45.517420 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:45.517436 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:45.517545 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:45.517829 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:45.517833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:45.518030 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:45.518150 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:45.518156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.519100 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.519106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:45.519749 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:45.519758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.520301 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.520305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:45.520403 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:45.521100 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:45.521145 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:45.523132 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
W20260619 06:58:45.523422 20887 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:45.517209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:51419 (local address 127.11.106.126:36423)
0619 06:58:45.517354 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:45.517359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:45.517515 (+ 156us) server_negotiation.cc:407] Connection header received
0619 06:58:45.517631 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:45.517633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:45.517687 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:45.517760 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:45.518281 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.518957 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.519906 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:45.520157 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:45.520290 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:45.523290 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:51419: BlockingRecv error: recv got EOF from 127.11.106.67:51419 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 06:58:46 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:46 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:46.150712 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:46.144312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:56397)
0619 06:58:46.144708 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:46.144723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:46.144837 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:46.145108 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:46.145110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:46.145304 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:46.145411 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:46.145417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.146262 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.146265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:46.146935 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:46.146946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.147419 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.147422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:46.147492 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:46.148135 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:46.148148 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:46.150547 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
W20260619 06:58:46.150964 20891 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:46.144395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:56397 (local address 127.11.106.126:36423)
0619 06:58:46.144727 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:46.144731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:46.144802 (+ 71us) server_negotiation.cc:407] Connection header received
0619 06:58:46.144928 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:46.144930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:46.144980 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:46.145068 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:46.145519 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.146126 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.147053 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.147312 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.147398 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:46.150796 (+ 3398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:56397: BlockingRecv error: recv got EOF from 127.11.106.65:56397 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":136,"threads_started":1}
Jun 19 06:58:46 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:46 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:46.314687 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:46.309120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:50279)
0619 06:58:46.309380 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:46.309394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:46.309495 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:46.309737 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:46.309739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:46.309914 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:46.310007 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:46.310013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.310757 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.310759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:46.311329 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:46.311337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.311785 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.311787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:46.311846 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:46.312390 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:46.312402 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:46.314506 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
W20260619 06:58:46.314730 20891 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:46.309171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:50279 (local address 127.11.106.126:36423)
0619 06:58:46.309316 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:46.309321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:46.309482 (+ 161us) server_negotiation.cc:407] Connection header received
0619 06:58:46.309563 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:46.309566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:46.309617 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:46.309696 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:46.310123 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.310662 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.311440 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.311686 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.311769 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:46.314630 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:50279: BlockingRecv error: recv got EOF from 127.11.106.66:50279 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 06:58:46 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:46 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:46.530311 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:46.524237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:50747)
0619 06:58:46.524578 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:46.524690 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:46.524816 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:46.525082 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:46.525084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:46.525269 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:46.525362 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:46.525367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.526135 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.526138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:46.526875 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:46.526888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.527421 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.527427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:46.527524 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:46.528114 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:46.528126 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:46.530138 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":119,"threads_started":1}
W20260619 06:58:46.530388 20891 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:46.524318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:50747 (local address 127.11.106.126:36423)
0619 06:58:46.524504 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:46.524511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:46.524802 (+ 291us) server_negotiation.cc:407] Connection header received
0619 06:58:46.524912 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:46.524914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:46.524965 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:46.525046 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:46.525511 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.526033 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.527042 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:46.527317 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:46.527402 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:46.530261 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:50747: BlockingRecv error: recv got EOF from 127.11.106.67:50747 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 19 06:58:47 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:47 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:47.157573 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:47.151785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:46595)
0619 06:58:47.152130 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:47.152147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:47.152279 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:47.152613 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:47.152616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:47.152865 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:47.152975 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:47.152981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.153786 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.153791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:47.154356 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:47.154363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.154836 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.154840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:47.154928 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:47.155437 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:47.155450 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:47.157459 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":95,"threads_started":1}
W20260619 06:58:47.157780 20895 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:47.151861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:46595 (local address 127.11.106.126:36423)
0619 06:58:47.152184 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:47.152191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:47.152208 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:58:47.152398 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:47.152401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:47.152456 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:47.152580 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:47.153112 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.153671 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.154477 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.154719 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.154832 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:47.157603 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:46595: BlockingRecv error: recv got EOF from 127.11.106.65:46595 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
Jun 19 06:58:47 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:47 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:47.321871 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:47.315560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:57547)
0619 06:58:47.315851 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:47.315865 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:47.315960 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:47.316211 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:47.316214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:47.316398 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:47.316494 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:47.316499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.317401 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.317404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:47.318003 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:47.318013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.318578 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.318582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:47.318681 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:47.319396 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:47.319410 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:47.321722 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":100,"threads_started":1}
W20260619 06:58:47.322091 20895 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:47.315687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:57547 (local address 127.11.106.126:36423)
0619 06:58:47.315854 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:47.315858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:47.315961 (+ 103us) server_negotiation.cc:407] Connection header received
0619 06:58:47.316031 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:47.316033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:47.316081 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:47.316169 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:47.316740 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.317291 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.318148 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.318448 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.318539 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:47.321926 (+ 3387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:57547: BlockingRecv error: recv got EOF from 127.11.106.66:57547 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 06:58:47 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:47 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:47.536885 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:47.531180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:48063)
0619 06:58:47.531464 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:47.531478 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:47.531572 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:47.531832 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:47.531835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:47.532018 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:47.532112 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:47.532116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.532919 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.532921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:47.533481 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:47.533489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.533892 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.533895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:47.533957 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:47.534723 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:47.534737 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:47.536755 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
W20260619 06:58:47.537091 20895 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:47.531336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:48063 (local address 127.11.106.126:36423)
0619 06:58:47.531480 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:47.531484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:47.531545 (+ 61us) server_negotiation.cc:407] Connection header received
0619 06:58:47.531666 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:47.531668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:47.531712 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:47.531787 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:47.532216 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.532821 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.533579 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:47.533788 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:47.533862 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:47.536928 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:48063: BlockingRecv error: recv got EOF from 127.11.106.67:48063 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 06:58:48 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:48 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:48.164080 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:48.158491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:36963)
0619 06:58:48.158793 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:48.158832 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:48.158963 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:48.159251 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:48.159253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:48.159448 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:48.159561 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:48.159569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.160382 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.160385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:48.161043 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:48.161054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.161505 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.161507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:48.161570 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:48.162123 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:48.162136 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:48.163947 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":104,"threads_started":1}
W20260619 06:58:48.164309 20899 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:48.158631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:36963 (local address 127.11.106.126:36423)
0619 06:58:48.158934 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:48.158940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:48.158956 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:48.159076 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:48.159078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:48.159123 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:48.159201 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:48.159691 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.160253 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.161174 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.161402 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.161484 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:48.164149 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:36963: BlockingRecv error: recv got EOF from 127.11.106.65:36963 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":63,"threads_started":1}
Jun 19 06:58:48 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:48 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:48.329246 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:48.322814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:36677)
0619 06:58:48.323133 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:48.323148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:48.323323 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:48.323682 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:48.323685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:48.323943 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:48.324100 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:48.324116 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.325452 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.325455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:48.326052 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:48.326062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.326508 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.326510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:48.326575 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:48.327128 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:48.327141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:48.329125 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":121,"threads_started":1}
W20260619 06:58:48.329619 20899 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:48.322953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:36677 (local address 127.11.106.126:36423)
0619 06:58:48.323114 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:48.323118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:48.323264 (+ 146us) server_negotiation.cc:407] Connection header received
0619 06:58:48.323443 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:48.323447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:48.323507 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:48.323627 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:48.324259 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.325095 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.326180 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.326404 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.326489 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:48.329301 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:36677: BlockingRecv error: recv got EOF from 127.11.106.66:36677 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 06:58:48 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:48 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:48.543154 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:48.537743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:35673)
0619 06:58:48.538035 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:48.538048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:48.538146 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:48.538376 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:48.538378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:48.538571 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:48.538712 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:48.538717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.539473 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.539476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:48.540095 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:48.540104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.540524 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.540526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:48.540705 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:48.541178 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:48.541190 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:48.543036 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
W20260619 06:58:48.543380 20899 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:48.537797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:35673 (local address 127.11.106.126:36423)
0619 06:58:48.537944 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:48.537948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:48.538113 (+ 165us) server_negotiation.cc:407] Connection header received
0619 06:58:48.538227 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:48.538229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:48.538275 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:48.538343 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:48.538824 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.539350 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.540210 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:48.540425 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:48.540502 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:48.543227 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:35673: BlockingRecv error: recv got EOF from 127.11.106.67:35673 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 06:58:49 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:49 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:49.171128 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:49.165017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:60677)
0619 06:58:49.165374 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:49.165387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:49.165488 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:49.165794 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:49.165796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:49.165977 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:49.166072 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:49.166077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.166881 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.166883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:49.167461 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:49.167470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.167998 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.168001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:49.168082 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:49.168724 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:49.168757 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:49.170953 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":94,"threads_started":1}
W20260619 06:58:49.171244 20903 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:49.165160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:60677 (local address 127.11.106.126:36423)
0619 06:58:49.165449 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:49.165453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:49.165469 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:58:49.165592 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:49.165596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:49.165653 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:49.165753 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:49.166180 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.166771 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.167582 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.167859 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.167948 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:49.171110 (+ 3162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:60677: BlockingRecv error: recv got EOF from 127.11.106.65:60677 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
Jun 19 06:58:49 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:49 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:49.335793 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:49.330115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:51291)
0619 06:58:49.330443 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:49.330464 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:49.330570 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:49.330812 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:49.330814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:49.331004 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:49.331103 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:49.331109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.331840 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.331844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:49.332418 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:49.332428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.332875 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.332878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:49.332945 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:49.333545 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:49.333559 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:49.335647 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":126,"threads_started":1}
W20260619 06:58:49.336016 20903 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:49.330176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:51291 (local address 127.11.106.126:36423)
0619 06:58:49.330332 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:49.330336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:49.330560 (+ 224us) server_negotiation.cc:407] Connection header received
0619 06:58:49.330646 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:49.330648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:49.330694 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:49.330766 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:49.331204 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.331715 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.332527 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.332765 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.332841 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:49.335877 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:51291: BlockingRecv error: recv got EOF from 127.11.106.66:51291 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:58:49 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:49 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:49.549924 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:49.544177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:51601)
0619 06:58:49.544519 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:49.544535 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:49.544672 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:49.544946 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:49.544950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:49.545167 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:49.545256 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:49.545261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.546047 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.546053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:49.546614 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:49.546621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.547093 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.547096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:49.547191 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:49.547756 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:49.547770 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:49.549755 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":122,"threads_started":1}
W20260619 06:58:49.549984 20903 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:49.544229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:51601 (local address 127.11.106.126:36423)
0619 06:58:49.544384 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:49.544390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:49.544621 (+ 231us) server_negotiation.cc:407] Connection header received
0619 06:58:49.544777 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:49.544781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:49.544832 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:49.544924 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:49.545381 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.545938 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.546740 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:49.546981 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:49.547089 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:49.549876 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:51601: BlockingRecv error: recv got EOF from 127.11.106.67:51601 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 06:58:50 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:50 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:50.181949 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:50.172053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:53529)
0619 06:58:50.172350 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:50.172363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:50.172460 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:50.172731 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:50.172734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:50.172915 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:50.173013 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:50.173018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.173766 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.173769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:50.174338 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:50.174347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.174764 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.174766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:50.174829 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:50.179850 (+ 5021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:50.179863 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:50.181832 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":117,"threads_started":1}
W20260619 06:58:50.182111 20907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:50.172134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:53529 (local address 127.11.106.126:36423)
0619 06:58:50.172376 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:50.172379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:50.172413 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:58:50.172561 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:50.172563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:50.172604 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:50.172690 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:50.173116 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.173642 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.174444 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.174660 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.174734 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:50.181975 (+ 7241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:53529: BlockingRecv error: recv got EOF from 127.11.106.65:53529 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
Jun 19 06:58:50 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:50 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:50.342417 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:50.336685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:34071)
0619 06:58:50.337023 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:50.337040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:50.337165 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:50.337448 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:50.337451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:50.337671 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:50.337759 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:50.337765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.338547 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.338552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:50.339128 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:50.339138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.339607 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.339611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:50.339689 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:50.340252 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:50.340274 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:50.342301 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
W20260619 06:58:50.342586 20907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:50.336858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:34071 (local address 127.11.106.126:36423)
0619 06:58:50.337027 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:50.337034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:50.337107 (+ 73us) server_negotiation.cc:407] Connection header received
0619 06:58:50.337278 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:50.337282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:50.337335 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:50.337425 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:50.337878 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.338423 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.339265 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.339490 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.339628 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:50.342443 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:34071: BlockingRecv error: recv got EOF from 127.11.106.66:34071 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 06:58:50 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:50 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:50.556811 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:50.550944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:52821)
0619 06:58:50.551319 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:50.551333 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:50.551476 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:50.551771 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:50.551774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:50.551986 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:50.552076 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:50.552082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.552981 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.552986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:50.553589 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:50.553598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.554091 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.554094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:50.554197 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:50.554773 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:50.554785 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:50.556692 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":95,"threads_started":1}
W20260619 06:58:50.556923 20907 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:50.550971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:52821 (local address 127.11.106.126:36423)
0619 06:58:50.551148 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:50.551153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:50.551422 (+ 269us) server_negotiation.cc:407] Connection header received
0619 06:58:50.551591 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:50.551594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:50.551656 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:50.551743 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:50.552210 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.552851 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.553729 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:50.553971 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:50.554084 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:50.556803 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:52821: BlockingRecv error: recv got EOF from 127.11.106.67:52821 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 06:58:51 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:51 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:51.188971 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:51.183063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:58263)
0619 06:58:51.183468 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:51.183485 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:51.183608 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:51.183934 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:51.183938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:51.184155 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:51.184256 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:51.184262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.185113 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.185118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:51.185705 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:51.185714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.186195 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.186199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:51.186290 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:51.186855 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:51.186867 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:51.188813 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":130,"threads_started":1}
W20260619 06:58:51.189098 20911 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:51.183125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:58263 (local address 127.11.106.126:36423)
0619 06:58:51.183492 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:51.183498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:51.183549 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:58:51.183756 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:51.183759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:51.183804 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:51.183910 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:51.184382 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.185000 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.185835 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.186072 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.186182 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:51.188964 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:58263: BlockingRecv error: recv got EOF from 127.11.106.65:58263 (error 108)
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":95,"threads_started":1}
Jun 19 06:58:51 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:51 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:51.348970 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:51.343313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:43865)
0619 06:58:51.343619 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:51.343633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:51.343741 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:51.343975 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:51.343977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:51.344152 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:51.344258 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:51.344263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.345221 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.345225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:51.345811 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:51.345820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.346240 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.346242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:51.346303 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:51.346898 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:51.346910 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:51.348829 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":116,"threads_started":1}
W20260619 06:58:51.349131 20911 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:51.343370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:43865 (local address 127.11.106.126:36423)
0619 06:58:51.343529 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:51.343534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:51.343731 (+ 197us) server_negotiation.cc:407] Connection header received
0619 06:58:51.343813 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:51.343815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:51.343862 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:51.343961 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:51.344366 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.345091 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.345919 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.346132 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.346233 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:51.348979 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:43865: BlockingRecv error: recv got EOF from 127.11.106.66:43865 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 06:58:51 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:51 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.67@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:51.563849 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:51.557669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.67:35369)
0619 06:58:51.557966 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:51.557979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:51.558076 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:51.558350 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:51.558353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:51.558537 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:51.558641 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:51.558646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.559464 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.559466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:51.560013 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:51.560021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.560493 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.560501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:51.560736 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:51.561438 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:51.561462 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:51.563695 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":104,"threads_started":1}
W20260619 06:58:51.563867 20911 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:51.557791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:35369 (local address 127.11.106.126:36423)
0619 06:58:51.557927 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:51.557931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:51.558046 (+ 115us) server_negotiation.cc:407] Connection header received
0619 06:58:51.558173 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:51.558176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:51.558221 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:51.558293 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:51.558744 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.559362 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.560110 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:51.560349 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:51.560421 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:51.563772 (+ 3351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.67:35369: BlockingRecv error: recv got EOF from 127.11.106.67:35369 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 06:58:52 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:52 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.65@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:52.196655 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:52.190009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.65:60075)
0619 06:58:52.190406 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:52.190443 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:52.190572 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:52.190889 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:52.190892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:52.191065 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:52.191163 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:52.191168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.191933 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.191937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:52.192684 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:52.192696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.193159 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.193162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:52.193268 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:52.193809 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:52.193823 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:52.196445 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":83,"threads_started":1}
W20260619 06:58:52.196949 20915 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:52.190134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:60075 (local address 127.11.106.126:36423)
0619 06:58:52.190388 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:52.190393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:52.190544 (+ 151us) server_negotiation.cc:407] Connection header received
0619 06:58:52.190667 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:52.190671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:52.190752 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:52.190841 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:52.191268 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.191805 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.192803 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.193026 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.193106 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:52.196668 (+ 3562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.65:60075: BlockingRecv error: recv got EOF from 127.11.106.65:60075 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
Jun 19 06:58:52 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:58:52 dist-test-slave-10d0 krb5kdc[19857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.106.66@KRBTEST.COM for oryx/127.11.106.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:58:52.355717 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:52.350189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36423 (local address 127.11.106.66:35725)
0619 06:58:52.350480 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:52.350503 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:52.350597 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:52.350851 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:52.350853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:52.351031 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:52.351121 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:52.351126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.352002 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.352005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:52.352594 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:52.352602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.353084 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.353088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:52.353174 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:52.353751 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:52.353763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:52.355534 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.106.126:36423: Server oryx/127.11.106.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
W20260619 06:58:52.355762 20915 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:52.350369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:35725 (local address 127.11.106.126:36423)
0619 06:58:52.350515 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:52.350519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:52.350554 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:58:52.350685 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:52.350687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:52.350731 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:52.350813 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:52.351269 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.351890 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.352725 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.352963 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.353077 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:52.355659 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.106.66:35725: BlockingRecv error: recv got EOF from 127.11.106.66:35725 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:58:52.446964 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 20440
I20260619 06:58:52.453002 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 20592
I20260619 06:58:52.458490 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 20724
I20260619 06:58:52.463807 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 20370
2026-06-19T06:58:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17964 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T06:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:52Z Disabled control of system clock
I20260619 06:58:52.494712 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45269
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:35117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:45269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:58:52.607038 20920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:52.607311 20920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:52.607344 20920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:52.611512 20920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:52.611572 20920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:52.611588 20920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:52.611604 20920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:52.611619 20920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:52.617125 20920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:35117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:45269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:52.618371 20920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:52.619966 20920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:52.628682 20925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:52.628710 20926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:52.628778 20928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:52.628839 20920 server_base.cc:1034] running on GCE node
I20260619 06:58:52.629572 20920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:52.630692 20920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:52.631891 20920 hybrid_clock.cc:648] HybridClock initialized: now 1781852332631874 us; error 31 us; skew 500 ppm
I20260619 06:58:52.633913 20920 webserver.cc:460] Webserver started at http://127.11.106.126:36643/ using document root <none> and password file <none>
I20260619 06:58:52.634541 20920 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:52.634640 20920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:52.634883 20920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:52.637058 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "09ff58ce6a21405aab7e3be52b289d6a"
format_stamp: "Formatted at 2026-06-19 06:58:52 on dist-test-slave-10d0"
server_key: "6a457a66360a273b820d4c5d1d29089e"
server_key_iv: "f485b22a762bd3f8b09147e8ae08d2fe"
server_key_version: "encryptionkey@0"
I20260619 06:58:52.637681 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "09ff58ce6a21405aab7e3be52b289d6a"
format_stamp: "Formatted at 2026-06-19 06:58:52 on dist-test-slave-10d0"
server_key: "6a457a66360a273b820d4c5d1d29089e"
server_key_iv: "f485b22a762bd3f8b09147e8ae08d2fe"
server_key_version: "encryptionkey@0"
I20260619 06:58:52.641378 20920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:58:52.643800 20934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:52.645144 20920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:52.645279 20920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "09ff58ce6a21405aab7e3be52b289d6a"
format_stamp: "Formatted at 2026-06-19 06:58:52 on dist-test-slave-10d0"
server_key: "6a457a66360a273b820d4c5d1d29089e"
server_key_iv: "f485b22a762bd3f8b09147e8ae08d2fe"
server_key_version: "encryptionkey@0"
I20260619 06:58:52.645391 20920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:52.659724 20920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:52.660485 20920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:52.660741 20920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:52.667665 20920 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:45269
I20260619 06:58:52.667732 20986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:45269 every 8 connection(s)
I20260619 06:58:52.668680 20920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:52.670454 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 20920
I20260619 06:58:52.670550 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:52.670949 11689 external_mini_cluster.cc:1428] Setting key 406f504c1c200d11a8276677370322b4
I20260619 06:58:52.672381 20987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:52.678105 20987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a: Bootstrap starting.
I20260619 06:58:52.680193 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:52.672391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.11.106.126:45269)
0619 06:58:52.673071 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:52.673086 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:52.673123 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:58:52.673773 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:52.673790 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:52.674182 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:52.674520 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:52.675006 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.676025 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.676995 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.677467 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.677644 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:52.678532 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:52.678537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:52.678549 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:52.678929 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:52.678981 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:52.679001 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:52.679112 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:52.679251 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 06:58:52.679496 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":153,"threads_started":1}
I20260619 06:58:52.680460 20987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:52.681274 20987 log.cc:824] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:52.683303 20987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a: No bootstrap required, opened a new log
I20260619 06:58:52.686823 20987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ff58ce6a21405aab7e3be52b289d6a" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45269 } }
I20260619 06:58:52.687032 20987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:52.687075 20987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ff58ce6a21405aab7e3be52b289d6a, State: Initialized, Role: FOLLOWER
I20260619 06:58:52.687479 20987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ff58ce6a21405aab7e3be52b289d6a" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45269 } }
I20260619 06:58:52.687602 20987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:52.687641 20987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:52.687709 20987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:52.688897 20987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ff58ce6a21405aab7e3be52b289d6a" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45269 } }
I20260619 06:58:52.689208 20987 leader_election.cc:302] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09ff58ce6a21405aab7e3be52b289d6a; no voters:
I20260619 06:58:52.689502 20987 leader_election.cc:288] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:52.689579 20992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:52.689767 20992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [term 1 LEADER]: Becoming Leader. State: Replica: 09ff58ce6a21405aab7e3be52b289d6a, State: Running, Role: LEADER
I20260619 06:58:52.690092 20992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ff58ce6a21405aab7e3be52b289d6a" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45269 } }
I20260619 06:58:52.690764 20987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:52.691928 20993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09ff58ce6a21405aab7e3be52b289d6a. Latest consensus state: current_term: 1 leader_uuid: "09ff58ce6a21405aab7e3be52b289d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ff58ce6a21405aab7e3be52b289d6a" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45269 } } }
I20260619 06:58:52.691931 20994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09ff58ce6a21405aab7e3be52b289d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09ff58ce6a21405aab7e3be52b289d6a" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45269 } } }
I20260619 06:58:52.692143 20993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:52.692188 20994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a [sys.catalog]: This master's current role is: LEADER
W20260619 06:58:52.695953 21008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:58:52.696062 21008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:58:52.696182 21001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:52.697010 21001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:52.702337 21001 catalog_manager.cc:1347] Generated new cluster ID: b87f7a78c95f47e18221271856d45dd8
I20260619 06:58:52.702404 21001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:52.714864 21001 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:52.716111 21001 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:52.727380 21001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09ff58ce6a21405aab7e3be52b289d6a: Generated new TSK 0
I20260619 06:58:52.728279 21001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:58:52.739811 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45269
--builtin_ntp_servers=127.11.106.84:35117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 06:58:52.856029 21011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:52.856323 21011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:52.856361 21011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:52.860498 21011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:52.860630 21011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:52.860692 21011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:52.865754 21011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:35117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:52.866961 21011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:52.868450 21011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:58:52.879050 21011 server_base.cc:1034] running on GCE node
W20260619 06:58:52.879012 21019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:52.879012 21017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:52.879019 21016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:52.879717 21011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:52.880352 21011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:52.881511 21011 hybrid_clock.cc:648] HybridClock initialized: now 1781852332881485 us; error 43 us; skew 500 ppm
I20260619 06:58:52.883307 21011 webserver.cc:460] Webserver started at http://127.11.106.65:33157/ using document root <none> and password file <none>
I20260619 06:58:52.883852 21011 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:52.883913 21011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:52.884076 21011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:52.886212 21011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "4ca3546890d446c1ab9a930b839e3528"
format_stamp: "Formatted at 2026-06-19 06:58:52 on dist-test-slave-10d0"
server_key: "1dd7f3e33615d4b46886ed414993ac82"
server_key_iv: "6679f4c065f9cbd3b2aac8aea113d49a"
server_key_version: "encryptionkey@0"
I20260619 06:58:52.886749 21011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "4ca3546890d446c1ab9a930b839e3528"
format_stamp: "Formatted at 2026-06-19 06:58:52 on dist-test-slave-10d0"
server_key: "1dd7f3e33615d4b46886ed414993ac82"
server_key_iv: "6679f4c065f9cbd3b2aac8aea113d49a"
server_key_version: "encryptionkey@0"
I20260619 06:58:52.890594 21011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:52.893370 21025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:52.894690 21011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:52.894801 21011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "4ca3546890d446c1ab9a930b839e3528"
format_stamp: "Formatted at 2026-06-19 06:58:52 on dist-test-slave-10d0"
server_key: "1dd7f3e33615d4b46886ed414993ac82"
server_key_iv: "6679f4c065f9cbd3b2aac8aea113d49a"
server_key_version: "encryptionkey@0"
I20260619 06:58:52.894892 21011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:52.925725 21011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:52.926441 21011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:52.926661 21011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:52.927253 21011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:52.928180 21011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:52.928228 21011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:52.928267 21011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:52.928283 21011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:52.937223 21011 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:40717
I20260619 06:58:52.937270 21138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:40717 every 8 connection(s)
I20260619 06:58:52.938220 21011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:52.947181 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 21011
I20260619 06:58:52.947285 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:52.947631 11689 external_mini_cluster.cc:1428] Setting key 37fdd9c91c3ffe9e42acc76b63b986a8
I20260619 06:58:52.948069 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:52.940175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:54253 (local address 127.11.106.126:45269)
0619 06:58:52.940320 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:52.940324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:52.941496 (+ 1172us) server_negotiation.cc:407] Connection header received
0619 06:58:52.942967 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:52.942971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:52.943028 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:52.943130 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:52.944048 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.944653 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.945845 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.946088 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.946207 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:52.947495 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:52.947499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:52.947501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:52.947513 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:52.947532 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:52.947540 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:52.947618 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:52.947841 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 06:58:52.947922 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:58:52.949133 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:52.940659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45269 (local address 127.11.106.65:54253)
0619 06:58:52.941341 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:52.941382 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:52.942739 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:52.943305 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:52.943312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:52.943401 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:52.943902 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:52.943914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.944781 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.944785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:52.945711 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:52.945721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:52.946254 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:52.946258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:52.946430 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:52.947174 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:52.947176 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:52.947220 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:52.947327 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:52.947633 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:52.947645 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:52.947651 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:52.947830 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 06:58:52.948108 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":127,"threads_started":1}
I20260619 06:58:52.949894 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45269
--builtin_ntp_servers=127.11.106.84:35117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:58:52.950513 21139 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45269
I20260619 06:58:52.950778 21139 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:52.951419 21139 heartbeater.cc:511] Master 127.11.106.126:45269 requested a full tablet report, sending...
I20260619 06:58:52.953063 20951 ts_manager.cc:194] Registered new tserver with Master: 4ca3546890d446c1ab9a930b839e3528 (127.11.106.65:40717)
I20260619 06:58:52.954288 20951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.65:54253
W20260619 06:58:53.067876 21142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:53.068142 21142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:53.068216 21142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:53.072326 21142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:53.072394 21142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:53.072449 21142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:53.077216 21142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:35117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:53.078351 21142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:53.079829 21142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:53.090020 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:53.090013 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:53.090072 21142 server_base.cc:1034] running on GCE node
W20260619 06:58:53.090013 21150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:53.090662 21142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:53.091363 21142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:53.092600 21142 hybrid_clock.cc:648] HybridClock initialized: now 1781852333092591 us; error 59 us; skew 500 ppm
I20260619 06:58:53.094619 21142 webserver.cc:460] Webserver started at http://127.11.106.66:38403/ using document root <none> and password file <none>
I20260619 06:58:53.095220 21142 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:53.095315 21142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:53.095548 21142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:53.097788 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "cf1b7fefe5b4482596093da843e8f933"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "05bcdc4deb6742ffbc073036db8630be"
server_key_iv: "47e0b2c5ad87f16d46e5b3407e800c38"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.098426 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "cf1b7fefe5b4482596093da843e8f933"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "05bcdc4deb6742ffbc073036db8630be"
server_key_iv: "47e0b2c5ad87f16d46e5b3407e800c38"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.102658 21142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:53.105474 21156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.106907 21142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:53.107076 21142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "cf1b7fefe5b4482596093da843e8f933"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "05bcdc4deb6742ffbc073036db8630be"
server_key_iv: "47e0b2c5ad87f16d46e5b3407e800c38"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.107219 21142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:53.117161 21142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:53.117873 21142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:53.118072 21142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:53.118686 21142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:53.119642 21142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:53.119711 21142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.119781 21142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:53.119828 21142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.129065 21142 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:40207
I20260619 06:58:53.129099 21269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:40207 every 8 connection(s)
I20260619 06:58:53.130014 21142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:53.136987 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 21142
I20260619 06:58:53.137115 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:53.137444 11689 external_mini_cluster.cc:1428] Setting key 2f96f667c14d68d5962d1a1cf1ac1a94
I20260619 06:58:53.139724 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45269
--builtin_ntp_servers=127.11.106.84:35117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:58:53.140443 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:53.131957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:43401 (local address 127.11.106.126:45269)
0619 06:58:53.132115 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:53.132120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:53.133324 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 06:58:53.134770 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:53.134773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:53.134834 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:53.134951 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:53.135879 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:53.136423 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:53.138131 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:53.138453 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:53.138561 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:53.139915 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:53.139922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:53.139926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:53.139944 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:53.139968 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:53.139977 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:53.140060 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:53.140262 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 06:58:53.140327 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:58:53.141804 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:53.132369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45269 (local address 127.11.106.66:43401)
0619 06:58:53.133149 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:53.133196 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:53.134544 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:53.135131 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:53.135137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:53.135227 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:53.135721 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:53.135734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:53.136586 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:53.136589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:53.137964 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:53.137974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:53.138658 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:53.138662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:53.138908 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:53.139566 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:53.139570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:53.139640 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:53.139728 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:53.140080 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:53.140093 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:53.140103 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:58:53.140248 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 06:58:53.140831 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":163,"threads_started":1}
I20260619 06:58:53.143056 21270 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45269
I20260619 06:58:53.143329 21270 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:53.143980 21270 heartbeater.cc:511] Master 127.11.106.126:45269 requested a full tablet report, sending...
I20260619 06:58:53.145298 20951 ts_manager.cc:194] Registered new tserver with Master: cf1b7fefe5b4482596093da843e8f933 (127.11.106.66:40207)
I20260619 06:58:53.146176 20951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.66:43401
W20260619 06:58:53.254866 21273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:53.255119 21273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:53.255154 21273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:53.259084 21273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:53.259150 21273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:53.259205 21273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:53.263903 21273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:35117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:53.264978 21273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:53.266322 21273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:53.275830 21278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:53.275920 21279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:53.275938 21281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:53.276309 21273 server_base.cc:1034] running on GCE node
I20260619 06:58:53.276775 21273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:53.277460 21273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:53.278662 21273 hybrid_clock.cc:648] HybridClock initialized: now 1781852333278642 us; error 28 us; skew 500 ppm
I20260619 06:58:53.280706 21273 webserver.cc:460] Webserver started at http://127.11.106.67:33219/ using document root <none> and password file <none>
I20260619 06:58:53.281363 21273 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:53.281464 21273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:53.281731 21273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:53.284034 21273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "e98dd9b7e05c493dbe3bca6fac96231c"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "85da88cd4d41ad431c78afdfb7c22c13"
server_key_iv: "4cb2bfc581db2ff38791edd3741a11cb"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.284742 21273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "e98dd9b7e05c493dbe3bca6fac96231c"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "85da88cd4d41ad431c78afdfb7c22c13"
server_key_iv: "4cb2bfc581db2ff38791edd3741a11cb"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.288895 21273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:58:53.291743 21287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.293253 21273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:58:53.293391 21273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "e98dd9b7e05c493dbe3bca6fac96231c"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "85da88cd4d41ad431c78afdfb7c22c13"
server_key_iv: "4cb2bfc581db2ff38791edd3741a11cb"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.293520 21273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:53.310544 21273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:53.311316 21273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:53.311527 21273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:53.312150 21273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:53.313174 21273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:53.313246 21273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.313321 21273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:53.313365 21273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.322994 21273 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:34589
I20260619 06:58:53.323028 21400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:34589 every 8 connection(s)
I20260619 06:58:53.323920 21273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:53.326992 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 21273
I20260619 06:58:53.327092 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:53.327405 11689 external_mini_cluster.cc:1428] Setting key aff0a2e7676b8769365285f59de80639
I20260619 06:58:53.333953 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:53.325943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:40099 (local address 127.11.106.126:45269)
0619 06:58:53.326089 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:53.326093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:53.327329 (+ 1236us) server_negotiation.cc:407] Connection header received
0619 06:58:53.329153 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:53.329156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:53.329211 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:53.329313 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:53.330268 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:53.330800 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:53.331962 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:53.332190 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:53.332303 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:53.333459 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:53.333463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:53.333465 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:53.333486 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:53.333504 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:53.333512 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:53.333592 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:53.333756 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 06:58:53.333818 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:58:53.334786 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:53.326370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45269 (local address 127.11.106.67:40099)
0619 06:58:53.327128 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:53.327209 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:53.328938 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:53.329494 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:53.329500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:53.329625 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:53.330119 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:53.330132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:53.330934 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:53.330937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:53.331830 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:53.331840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:53.332342 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:53.332346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:53.332577 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:53.333205 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:53.333207 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:53.333246 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:53.333307 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:53.333595 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:53.333607 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:53.333613 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:53.333754 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 06:58:53.334028 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260619 06:58:53.335942 21401 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45269
I20260619 06:58:53.336153 21401 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:53.336732 21401 heartbeater.cc:511] Master 127.11.106.126:45269 requested a full tablet report, sending...
I20260619 06:58:53.337775 20951 ts_manager.cc:194] Registered new tserver with Master: e98dd9b7e05c493dbe3bca6fac96231c (127.11.106.67:34589)
I20260619 06:58:53.338366 20951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.67:40099
I20260619 06:58:53.341284 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:58:53.344252 20990 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:53.343210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.11.106.126:45269)
0619 06:58:53.343370 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:53.343375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:53.343579 (+ 204us) server_negotiation.cc:407] Connection header received
0619 06:58:53.343618 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:53.343620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:53.343661 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:53.343733 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:53.344134 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58422: BlockingRecv error: recv got EOF from 127.0.0.1:58422 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:58:53.344295 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 21011
I20260619 06:58:53.350749 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 21142
I20260619 06:58:53.356215 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 21273
I20260619 06:58:53.361888 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 20920
2026-06-19T06:58:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (897 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T06:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:53Z Disabled control of system clock
I20260619 06:58:53.391894 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46675
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:46675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:58:53.503789 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:53.504036 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:53.504072 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:53.508019 21412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:53.508070 21412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:53.508085 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:53.508102 21412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:53.508117 21412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:53.513012 21412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:46675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:53.514005 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:53.515321 21412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:53.523969 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:53.524238 21412 server_base.cc:1034] running on GCE node
W20260619 06:58:53.523986 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:53.523972 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:53.524991 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:53.526117 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:53.527297 21412 hybrid_clock.cc:648] HybridClock initialized: now 1781852333527269 us; error 44 us; skew 500 ppm
I20260619 06:58:53.529305 21412 webserver.cc:460] Webserver started at http://127.11.106.126:42231/ using document root <none> and password file <none>
I20260619 06:58:53.529875 21412 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:53.529937 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:53.530182 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:53.532228 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "7b09d308e2d54f4c8da3076612407814"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "7e9251ec34e92bba9707d853fedcf25b"
server_key_iv: "03a5f4b2f5c1d3ea5f848879611faa34"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.532812 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "7b09d308e2d54f4c8da3076612407814"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "7e9251ec34e92bba9707d853fedcf25b"
server_key_iv: "03a5f4b2f5c1d3ea5f848879611faa34"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.536334 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:58:53.538873 21426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.540096 21412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:53.540237 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "7b09d308e2d54f4c8da3076612407814"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "7e9251ec34e92bba9707d853fedcf25b"
server_key_iv: "03a5f4b2f5c1d3ea5f848879611faa34"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.540349 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:53.544783 21412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:53.545393 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:53.545594 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:53.552484 21412 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:46675
I20260619 06:58:53.552515 21478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:46675 every 8 connection(s)
I20260619 06:58:53.553467 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:53.556742 21479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:53.557468 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 21412
I20260619 06:58:53.557608 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:53.557917 11689 external_mini_cluster.cc:1428] Setting key 54b87bc61ec30190bd2df279d4f6d871
I20260619 06:58:53.563429 21479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814: Bootstrap starting.
I20260619 06:58:53.563824 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:53.559281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.11.106.126:46675)
0619 06:58:53.559939 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:53.559956 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:53.559994 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:58:53.560721 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:53.560743 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:53.561131 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:53.561438 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:53.562355 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:53.562359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:53.562372 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:53.562383 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:53.562442 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:53.562458 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:53.562574 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:53.562703 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 06:58:53.562999 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":152,"threads_started":1}
I20260619 06:58:53.566087 21479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:53.566916 21479 log.cc:824] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:53.568972 21479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814: No bootstrap required, opened a new log
I20260619 06:58:53.572911 21479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09d308e2d54f4c8da3076612407814" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46675 } }
I20260619 06:58:53.573230 21479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:53.573345 21479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b09d308e2d54f4c8da3076612407814, State: Initialized, Role: FOLLOWER
I20260619 06:58:53.573784 21479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09d308e2d54f4c8da3076612407814" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46675 } }
I20260619 06:58:53.573912 21479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:53.573961 21479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:53.574033 21479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:53.574932 21479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09d308e2d54f4c8da3076612407814" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46675 } }
I20260619 06:58:53.575248 21479 leader_election.cc:302] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b09d308e2d54f4c8da3076612407814; no voters:
I20260619 06:58:53.575531 21479 leader_election.cc:288] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:53.575659 21484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:53.575940 21484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [term 1 LEADER]: Becoming Leader. State: Replica: 7b09d308e2d54f4c8da3076612407814, State: Running, Role: LEADER
I20260619 06:58:53.576289 21484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09d308e2d54f4c8da3076612407814" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46675 } }
I20260619 06:58:53.576961 21479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:53.578192 21486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b09d308e2d54f4c8da3076612407814. Latest consensus state: current_term: 1 leader_uuid: "7b09d308e2d54f4c8da3076612407814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09d308e2d54f4c8da3076612407814" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46675 } } }
I20260619 06:58:53.578341 21486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:53.578622 21485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b09d308e2d54f4c8da3076612407814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09d308e2d54f4c8da3076612407814" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46675 } } }
I20260619 06:58:53.578711 21485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:53.578747 21493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:53.582002 21493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:53.587966 21493 catalog_manager.cc:1347] Generated new cluster ID: aed822f6b2624467b50cebb3a3969ef1
I20260619 06:58:53.588033 21493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:53.605410 21493 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:53.605753 21493 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:53.620821 21493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b09d308e2d54f4c8da3076612407814: Generated new TSK 0
I20260619 06:58:53.621706 21493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:58:53.628379 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46675
--builtin_ntp_servers=127.11.106.84:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:58:53.738193 21503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:53.738520 21503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:53.738597 21503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:53.742611 21503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:53.742708 21503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:53.742800 21503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:53.747740 21503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:53.748886 21503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:53.750272 21503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:53.759827 21509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:53.759917 21511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:53.759827 21508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:53.760042 21503 server_base.cc:1034] running on GCE node
I20260619 06:58:53.760501 21503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:53.761181 21503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:53.762377 21503 hybrid_clock.cc:648] HybridClock initialized: now 1781852333762360 us; error 27 us; skew 500 ppm
I20260619 06:58:53.764122 21503 webserver.cc:460] Webserver started at http://127.11.106.65:40437/ using document root <none> and password file <none>
I20260619 06:58:53.764763 21503 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:53.764854 21503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:53.765079 21503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:53.767066 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "137b999735bd4ded925cc1cffc704d46"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "e168a9c163e7a98326de902cba838d3c"
server_key_iv: "30234e58ee68eb5c548cc56dc20860e4"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.767618 21503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "137b999735bd4ded925cc1cffc704d46"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "e168a9c163e7a98326de902cba838d3c"
server_key_iv: "30234e58ee68eb5c548cc56dc20860e4"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.771147 21503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 06:58:53.773727 21517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.774881 21503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:58:53.775009 21503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "137b999735bd4ded925cc1cffc704d46"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "e168a9c163e7a98326de902cba838d3c"
server_key_iv: "30234e58ee68eb5c548cc56dc20860e4"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.775117 21503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:53.779865 21503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:53.780444 21503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:53.780645 21503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:53.781172 21503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:53.782060 21503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:53.782130 21503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.782197 21503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:53.782243 21503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.790813 21503 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:33267
I20260619 06:58:53.790851 21630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:33267 every 8 connection(s)
I20260619 06:58:53.791697 21503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:53.794596 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 21503
I20260619 06:58:53.794714 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:53.795056 11689 external_mini_cluster.cc:1428] Setting key cb4283eb49cd83a90cf4ba0690a9a716
I20260619 06:58:53.797217 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46675
--builtin_ntp_servers=127.11.106.84:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 06:58:53.798410 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:53.793547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:45249 (local address 127.11.106.126:46675)
0619 06:58:53.793717 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:53.793722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:53.794790 (+ 1068us) server_negotiation.cc:407] Connection header received
0619 06:58:53.796396 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:53.796399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:53.796441 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:53.796532 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:53.797893 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:53.797896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:53.797898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:53.797909 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:53.797928 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:53.797936 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:53.798014 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:53.798256 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 06:58:53.798297 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 06:58:53.799325 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:53.793867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46675 (local address 127.11.106.65:45249)
0619 06:58:53.794609 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:53.794666 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:53.796164 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:53.796744 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:53.796753 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:53.796865 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:53.797569 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:53.797574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:53.797631 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:53.797724 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:53.798030 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:53.798041 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:53.798050 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:58:53.798230 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 06:58:53.798478 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260619 06:58:53.800657 21631 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46675
I20260619 06:58:53.800886 21631 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:53.801312 21631 heartbeater.cc:511] Master 127.11.106.126:46675 requested a full tablet report, sending...
I20260619 06:58:53.803041 21443 ts_manager.cc:194] Registered new tserver with Master: 137b999735bd4ded925cc1cffc704d46 (127.11.106.65:33267)
W20260619 06:58:53.909822 21634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:53.910084 21634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:53.910131 21634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:53.914047 21634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:53.914094 21634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:53.914155 21634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:53.918887 21634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:53.919970 21634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:53.921494 21634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:58:53.931454 21634 server_base.cc:1034] running on GCE node
W20260619 06:58:53.931404 21639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:53.931404 21640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:53.931398 21642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:53.932152 21634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:53.932919 21634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:53.934092 21634 hybrid_clock.cc:648] HybridClock initialized: now 1781852333934057 us; error 54 us; skew 500 ppm
I20260619 06:58:53.936033 21634 webserver.cc:460] Webserver started at http://127.11.106.66:42671/ using document root <none> and password file <none>
I20260619 06:58:53.936699 21634 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:53.936767 21634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:53.936977 21634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:53.939164 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "00e1b808900949699e14f4f3b943459a"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "430247c86c29e678999e4602ce73c996"
server_key_iv: "eda58a4cc40d9eebc4f3ae914e9b5c94"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.939733 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "00e1b808900949699e14f4f3b943459a"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "430247c86c29e678999e4602ce73c996"
server_key_iv: "eda58a4cc40d9eebc4f3ae914e9b5c94"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.943440 21634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 06:58:53.946282 21648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.947500 21634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 06:58:53.947636 21634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "00e1b808900949699e14f4f3b943459a"
format_stamp: "Formatted at 2026-06-19 06:58:53 on dist-test-slave-10d0"
server_key: "430247c86c29e678999e4602ce73c996"
server_key_iv: "eda58a4cc40d9eebc4f3ae914e9b5c94"
server_key_version: "encryptionkey@0"
I20260619 06:58:53.947796 21634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:53.952296 21634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:53.952930 21634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:53.953100 21634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:53.953622 21634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:53.954499 21634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:53.954543 21634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.954579 21634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:53.954594 21634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:53.963114 21634 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:43967
I20260619 06:58:53.963174 21761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:43967 every 8 connection(s)
I20260619 06:58:53.964157 21634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:58:53.970387 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:53.966130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:50765 (local address 127.11.106.126:46675)
0619 06:58:53.966310 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:53.966315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:53.967409 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 06:58:53.968729 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:53.968732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:53.968771 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:53.968854 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:53.969991 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:53.969995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:53.969997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:53.970008 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:53.970027 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:53.970035 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:53.970118 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:53.970240 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 06:58:53.970280 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:58:53.971256 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:53.966581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46675 (local address 127.11.106.66:50765)
0619 06:58:53.967263 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:53.967303 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:53.968471 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:53.969020 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:53.969027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:53.969114 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:53.969706 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:53.969708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:53.969766 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:53.969832 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:53.970117 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:53.970123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:53.970128 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:58:53.970254 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 06:58:53.970470 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":121,"threads_started":1}
I20260619 06:58:53.972410 21762 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46675
I20260619 06:58:53.972692 21762 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:53.973104 21762 heartbeater.cc:511] Master 127.11.106.126:46675 requested a full tablet report, sending...
I20260619 06:58:53.974105 21443 ts_manager.cc:194] Registered new tserver with Master: 00e1b808900949699e14f4f3b943459a (127.11.106.66:43967)
I20260619 06:58:53.974244 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 21634
I20260619 06:58:53.974371 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:53.974743 11689 external_mini_cluster.cc:1428] Setting key 69286de24603cc52b3b46c28e459e3bc
I20260619 06:58:53.976655 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46675
--builtin_ntp_servers=127.11.106.84:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:58:54.090123 21765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:54.090370 21765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:54.090408 21765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:54.094323 21765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:54.094390 21765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:54.094444 21765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:54.099121 21765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:54.100190 21765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:54.101722 21765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:54.111415 21770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:54.111454 21771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:54.111490 21773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:54.111778 21765 server_base.cc:1034] running on GCE node
I20260619 06:58:54.112313 21765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:54.113023 21765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:54.114192 21765 hybrid_clock.cc:648] HybridClock initialized: now 1781852334114178 us; error 25 us; skew 500 ppm
I20260619 06:58:54.115993 21765 webserver.cc:460] Webserver started at http://127.11.106.67:33275/ using document root <none> and password file <none>
I20260619 06:58:54.116657 21765 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:54.116724 21765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:54.116956 21765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:54.119006 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "3f3a08c505794ee291b9a5f300fbfae3"
format_stamp: "Formatted at 2026-06-19 06:58:54 on dist-test-slave-10d0"
server_key: "b5b9b33844e399c5713cd635cde7b279"
server_key_iv: "7305ff2f642d7aeebb3eb26df6cdb043"
server_key_version: "encryptionkey@0"
I20260619 06:58:54.119637 21765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "3f3a08c505794ee291b9a5f300fbfae3"
format_stamp: "Formatted at 2026-06-19 06:58:54 on dist-test-slave-10d0"
server_key: "b5b9b33844e399c5713cd635cde7b279"
server_key_iv: "7305ff2f642d7aeebb3eb26df6cdb043"
server_key_version: "encryptionkey@0"
I20260619 06:58:54.123624 21765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 06:58:54.126281 21779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:54.127504 21765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:54.127636 21765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "3f3a08c505794ee291b9a5f300fbfae3"
format_stamp: "Formatted at 2026-06-19 06:58:54 on dist-test-slave-10d0"
server_key: "b5b9b33844e399c5713cd635cde7b279"
server_key_iv: "7305ff2f642d7aeebb3eb26df6cdb043"
server_key_version: "encryptionkey@0"
I20260619 06:58:54.127753 21765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:54.132220 21765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:54.132879 21765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:54.133054 21765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:54.133575 21765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:54.134447 21765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:54.134517 21765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:54.134578 21765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:54.134625 21765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:54.143306 21765 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:36157
I20260619 06:58:54.143334 21892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:36157 every 8 connection(s)
I20260619 06:58:54.144229 21765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:54.150452 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:54.146111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:49393 (local address 127.11.106.126:46675)
0619 06:58:54.146265 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:54.146269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:54.147351 (+ 1082us) server_negotiation.cc:407] Connection header received
0619 06:58:54.148733 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:54.148736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:54.148773 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:54.148851 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:54.149938 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:54.149941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:54.149943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:58:54.149955 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:54.149975 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:58:54.149983 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:54.150066 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:54.150292 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 06:58:54.150337 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:58:54.151388 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:54.146557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46675 (local address 127.11.106.67:49393)
0619 06:58:54.147229 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:54.147268 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:54.148457 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:54.149008 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:54.149015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:54.149105 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:54.149668 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:58:54.149670 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:54.149710 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:58:54.149780 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:54.150105 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:54.150120 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:54.150126 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:58:54.150284 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 06:58:54.150586 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260619 06:58:54.152666 21893 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46675
I20260619 06:58:54.152920 21893 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:54.153064 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 21765
I20260619 06:58:54.153183 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:54.153383 21893 heartbeater.cc:511] Master 127.11.106.126:46675 requested a full tablet report, sending...
I20260619 06:58:54.153481 11689 external_mini_cluster.cc:1428] Setting key 9f9399126ec9b3ef5b16fc1fe7cd9853
I20260619 06:58:54.154412 21442 ts_manager.cc:194] Registered new tserver with Master: 3f3a08c505794ee291b9a5f300fbfae3 (127.11.106.67:36157)
I20260619 06:58:54.155864 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:58:54.159010 21482 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:58:54.157995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.11.106.126:46675)
0619 06:58:54.158163 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:54.158169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:54.158323 (+ 154us) server_negotiation.cc:407] Connection header received
0619 06:58:54.158494 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:54.158498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:54.158548 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:54.158650 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:54.158891 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34506: BlockingRecv error: recv got EOF from 127.0.0.1:34506 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:58:54.159682 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 21503
I20260619 06:58:54.165818 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 21634
I20260619 06:58:54.171420 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 21765
I20260619 06:58:54.176879 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 21412
2026-06-19T06:58:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (815 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:54 dist-test-slave-10d0 krb5kdc[21903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:54 dist-test-slave-10d0 krb5kdc[21903](info): set up 2 sockets
Jun 19 06:58:54 dist-test-slave-10d0 krb5kdc[21903](info): commencing operation
krb5kdc: starting...
W20260619 06:58:55.222194 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:55 dist-test-slave-10d0 krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:58:55.344069 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42087
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:42087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:55.453691 21916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:55.453979 21916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:55.454022 21916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:55.458117 21916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:58:55.458181 21916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:55.458207 21916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:55.458225 21916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:58:55.458240 21916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:58:55.463625 21916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:42087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:55.464689 21916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:55.465682 21916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:55.474466 21921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:55.474502 21924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:55.474521 21916 server_base.cc:1034] running on GCE node
W20260619 06:58:55.474485 21922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:55.475423 21916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:55.476639 21916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:55.477833 21916 hybrid_clock.cc:648] HybridClock initialized: now 1781852335477817 us; error 32 us; skew 500 ppm
Jun 19 06:58:55 dist-test-slave-10d0 krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:55.486184 21916 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:58:55.487129 21916 webserver.cc:460] Webserver started at http://127.11.106.126:39625/ using document root <none> and password file <none>
I20260619 06:58:55.487848 21916 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:55.488016 21916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:55.488287 21916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:55.490553 21916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "3cc656e21fc54f27809758cff7ecf557"
format_stamp: "Formatted at 2026-06-19 06:58:55 on dist-test-slave-10d0"
server_key: "8388138bbebc9c32334f3c6bfdbf642e"
server_key_iv: "13528585e113ab4c3c84765ae7a09ade"
server_key_version: "encryptionkey@0"
I20260619 06:58:55.491134 21916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "3cc656e21fc54f27809758cff7ecf557"
format_stamp: "Formatted at 2026-06-19 06:58:55 on dist-test-slave-10d0"
server_key: "8388138bbebc9c32334f3c6bfdbf642e"
server_key_iv: "13528585e113ab4c3c84765ae7a09ade"
server_key_version: "encryptionkey@0"
I20260619 06:58:55.495163 21916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:58:55.498037 21931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:55.499482 21916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:58:55.499611 21916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "3cc656e21fc54f27809758cff7ecf557"
format_stamp: "Formatted at 2026-06-19 06:58:55 on dist-test-slave-10d0"
server_key: "8388138bbebc9c32334f3c6bfdbf642e"
server_key_iv: "13528585e113ab4c3c84765ae7a09ade"
server_key_version: "encryptionkey@0"
I20260619 06:58:55.499747 21916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:55.529009 21916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:55.531921 21916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:55.532166 21916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:55.539172 21916 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:42087
I20260619 06:58:55.539467 21983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:42087 every 8 connection(s)
I20260619 06:58:55.540213 21916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:58:55.543095 21984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:55.548799 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557: Bootstrap starting.
I20260619 06:58:55.550406 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 21916
I20260619 06:58:55.550511 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:58:55.550843 11689 external_mini_cluster.cc:1428] Setting key a9a239a19496b61819651641d7954e04
I20260619 06:58:55.551563 21984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:55.552479 21984 log.cc:824] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:55.554471 21984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557: No bootstrap required, opened a new log
I20260619 06:58:55.558068 21984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc656e21fc54f27809758cff7ecf557" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42087 } }
I20260619 06:58:55.558374 21984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:55.558461 21984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc656e21fc54f27809758cff7ecf557, State: Initialized, Role: FOLLOWER
I20260619 06:58:55.558985 21984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc656e21fc54f27809758cff7ecf557" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42087 } }
I20260619 06:58:55.559139 21984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:58:55.559204 21984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:58:55.559301 21984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 06:58:55 dist-test-slave-10d0 krb5kdc[21903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:55.560283 21984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc656e21fc54f27809758cff7ecf557" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42087 } }
I20260619 06:58:55.560690 21984 leader_election.cc:302] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cc656e21fc54f27809758cff7ecf557; no voters:
I20260619 06:58:55.561082 21984 leader_election.cc:288] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:58:55.561151 21989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:55.561609 21989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [term 1 LEADER]: Becoming Leader. State: Replica: 3cc656e21fc54f27809758cff7ecf557, State: Running, Role: LEADER
I20260619 06:58:55.561903 21989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc656e21fc54f27809758cff7ecf557" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42087 } }
I20260619 06:58:55.562424 21984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:58:55.563165 21991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cc656e21fc54f27809758cff7ecf557. Latest consensus state: current_term: 1 leader_uuid: "3cc656e21fc54f27809758cff7ecf557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc656e21fc54f27809758cff7ecf557" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42087 } } }
I20260619 06:58:55.563290 21991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:55.563973 21990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cc656e21fc54f27809758cff7ecf557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc656e21fc54f27809758cff7ecf557" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42087 } } }
I20260619 06:58:55.564070 21990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557 [sys.catalog]: This master's current role is: LEADER
I20260619 06:58:55.564339 21997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:58:55.566326 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:55.552019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.11.106.126:42087)
0619 06:58:55.552488 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:55.552504 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:55.552608 (+ 104us) server_negotiation.cc:407] Connection header received
0619 06:58:55.553361 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:55.553379 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:55.553728 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:55.554081 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:55.554920 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:55.555914 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:55.556820 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:55.557318 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:55.557497 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:55.560252 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:55.560258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:55.560272 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:55.560299 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:55.562347 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:55.562831 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:55.562838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:55.562847 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:55.562995 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:55.563359 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:55.563363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:55.563366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:55.563903 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:55.564129 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:55.565003 (+ 874us) server_negotiation.cc:299] Negotiation successful
0619 06:58:55.565319 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":152,"threads_started":1}
I20260619 06:58:55.568032 21997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:58:55.573619 21997 catalog_manager.cc:1347] Generated new cluster ID: f3ed6ceca28944daabb96045e71acd68
I20260619 06:58:55.573696 21997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:58:55.579442 21997 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:58:55.580706 21997 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:58:55.589638 21997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557: Generated new TSK 0
I20260619 06:58:55.590443 21997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:58:55.630059 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42087
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:55.742130 22010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:55.742393 22010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:55.742435 22010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:55.746472 22010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:55.746528 22010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:55.746623 22010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:58:55.751816 22010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:55.752929 22010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:55.753964 22010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:55.764016 22018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:55.764081 22015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:55.764222 22010 server_base.cc:1034] running on GCE node
W20260619 06:58:55.764017 22016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:55.764793 22010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:55.765467 22010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:55.766662 22010 hybrid_clock.cc:648] HybridClock initialized: now 1781852335766635 us; error 48 us; skew 500 ppm
Jun 19 06:58:55 dist-test-slave-10d0 krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:55.769466 22010 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:58:55.770339 22010 webserver.cc:460] Webserver started at http://127.11.106.65:38661/ using document root <none> and password file <none>
I20260619 06:58:55.770938 22010 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:55.771034 22010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:55.771282 22010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:55.773406 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "8ebbd8f8892645a897253c13d7ec0e54"
format_stamp: "Formatted at 2026-06-19 06:58:55 on dist-test-slave-10d0"
server_key: "6c70f8d5ce029bceaa121201ff0894fc"
server_key_iv: "0fd77c3ab39793bdbc899fca511a78e3"
server_key_version: "encryptionkey@0"
I20260619 06:58:55.773963 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "8ebbd8f8892645a897253c13d7ec0e54"
format_stamp: "Formatted at 2026-06-19 06:58:55 on dist-test-slave-10d0"
server_key: "6c70f8d5ce029bceaa121201ff0894fc"
server_key_iv: "0fd77c3ab39793bdbc899fca511a78e3"
server_key_version: "encryptionkey@0"
I20260619 06:58:55.777645 22010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:58:55.780062 22025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:55.781402 22010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:58:55.781550 22010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "8ebbd8f8892645a897253c13d7ec0e54"
format_stamp: "Formatted at 2026-06-19 06:58:55 on dist-test-slave-10d0"
server_key: "6c70f8d5ce029bceaa121201ff0894fc"
server_key_iv: "0fd77c3ab39793bdbc899fca511a78e3"
server_key_version: "encryptionkey@0"
I20260619 06:58:55.781662 22010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:55.803879 22010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:55.807731 22010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:55.807945 22010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:55.808647 22010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:55.809518 22010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:55.809594 22010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:55.809660 22010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:55.809715 22010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:55.819286 22010 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:43873
I20260619 06:58:55.819304 22138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:43873 every 8 connection(s)
I20260619 06:58:55.820258 22010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:58:55.826969 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 22010
I20260619 06:58:55.827100 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:58:55.827402 11689 external_mini_cluster.cc:1428] Setting key 465ad2ffe428b1e48038382bd522bed6
Jun 19 06:58:55 dist-test-slave-10d0 krb5kdc[21903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:55.835544 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:55.822160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:57219 (local address 127.11.106.126:42087)
0619 06:58:55.822297 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:55.822301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:55.823184 (+ 883us) server_negotiation.cc:407] Connection header received
0619 06:58:55.824702 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:55.824706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:55.824785 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:55.824865 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:55.826041 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:55.826578 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:55.827831 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:55.828114 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:55.828228 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:55.831266 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:55.831272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:55.831278 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:55.831311 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:55.833106 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:55.833635 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:55.833642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:55.833647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:55.833761 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:55.834115 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:55.834121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:55.834123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:55.834387 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:55.834504 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:55.835114 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 06:58:55.835282 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:58:55.836408 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:55.822578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42087 (local address 127.11.106.65:57219)
0619 06:58:55.823027 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:55.823071 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:55.824402 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:55.825038 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:55.825046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:55.825414 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:55.825898 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:55.825912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:55.826687 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:55.826690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:55.827691 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:55.827700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:55.828280 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:55.828284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:55.828472 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:55.829277 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:55.829293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:55.831081 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:55.833233 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:55.833238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:55.833253 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:55.833525 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:55.833873 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:55.833875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:55.833877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:55.834019 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:55.834528 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:55.834534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:55.834845 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 06:58:55.835024 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 06:58:55.835320 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 06:58:55.837911 22139 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42087
I20260619 06:58:55.838181 22139 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:55.838730 22139 heartbeater.cc:511] Master 127.11.106.126:42087 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 06:58:55.840552 21948 ts_manager.cc:194] Registered new tserver with Master: 8ebbd8f8892645a897253c13d7ec0e54 (127.11.106.65:43873)
I20260619 06:58:55.841814 21948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:57219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:58:55.856206 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42087
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:55.967850 22144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:55.968163 22144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:55.968264 22144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:55.972342 22144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:55.972438 22144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:55.972548 22144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:58:55.977865 22144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:55.979018 22144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:55.980087 22144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:58:55.989916 22144 server_base.cc:1034] running on GCE node
W20260619 06:58:55.989825 22150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:55.989837 22152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:55.989840 22149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:55.990522 22144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:55.991214 22144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:55.992409 22144 hybrid_clock.cc:648] HybridClock initialized: now 1781852335992375 us; error 47 us; skew 500 ppm
Jun 19 06:58:55 dist-test-slave-10d0 krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:55.995304 22144 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:58:55.996199 22144 webserver.cc:460] Webserver started at http://127.11.106.66:42359/ using document root <none> and password file <none>
I20260619 06:58:55.996898 22144 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:55.996995 22144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:55.997231 22144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:55.999275 22144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b"
format_stamp: "Formatted at 2026-06-19 06:58:55 on dist-test-slave-10d0"
server_key: "d8d71d77b755504dc5ccd54305b319e5"
server_key_iv: "dc71e5133a12c5117cb5a5718701331d"
server_key_version: "encryptionkey@0"
I20260619 06:58:55.999842 22144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b"
format_stamp: "Formatted at 2026-06-19 06:58:55 on dist-test-slave-10d0"
server_key: "d8d71d77b755504dc5ccd54305b319e5"
server_key_iv: "dc71e5133a12c5117cb5a5718701331d"
server_key_version: "encryptionkey@0"
I20260619 06:58:56.003707 22144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:58:56.006341 22159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:56.007618 22144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:56.007844 22144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b"
format_stamp: "Formatted at 2026-06-19 06:58:55 on dist-test-slave-10d0"
server_key: "d8d71d77b755504dc5ccd54305b319e5"
server_key_iv: "dc71e5133a12c5117cb5a5718701331d"
server_key_version: "encryptionkey@0"
I20260619 06:58:56.007987 22144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:56.027067 22144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:56.030289 22144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:56.030512 22144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:56.031160 22144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:56.032078 22144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:56.032150 22144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:56.032217 22144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:56.032262 22144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:56.042186 22144 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:45269
I20260619 06:58:56.042208 22272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:45269 every 8 connection(s)
I20260619 06:58:56.043143 22144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
Jun 19 06:58:56 dist-test-slave-10d0 krb5kdc[21903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852335, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:56.053380 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 22144
I20260619 06:58:56.053499 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:58:56.053828 11689 external_mini_cluster.cc:1428] Setting key f2fd375d9d7f7a67efe6ff692f9933cf
I20260619 06:58:56.058327 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.045147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:41575 (local address 127.11.106.126:42087)
0619 06:58:56.045296 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:56.045300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:56.046157 (+ 857us) server_negotiation.cc:407] Connection header received
0619 06:58:56.047622 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:56.047626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:56.047698 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:56.047799 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:56.049063 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.049601 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.050773 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.051005 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.051119 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.053954 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:56.053960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:56.053963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:56.053995 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:56.055866 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:56.056384 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:56.056391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:56.056396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:56.056493 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:56.056913 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:56.056919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:56.056924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:56.057184 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:56.057294 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:56.058065 (+ 771us) server_negotiation.cc:299] Negotiation successful
0619 06:58:56.058189 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:58:56.059363 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.045554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42087 (local address 127.11.106.66:41575)
0619 06:58:56.046003 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:56.046048 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:56.047390 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:56.047982 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:56.047989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:56.048370 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:56.048916 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.048930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.049758 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.049761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.050643 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.050651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.051161 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.051165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.051339 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.051949 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:56.051966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:56.053761 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:56.055992 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:56.055997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:56.056010 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:56.056278 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:56.056639 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:56.056641 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:56.056643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:56.056809 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:56.057353 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:56.057361 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:56.057770 (+ 409us) client_negotiation.cc:769] Sending connection context
0619 06:58:56.058016 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 06:58:56.058427 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":98,"threads_started":1}
I20260619 06:58:56.060734 22273 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42087
I20260619 06:58:56.061002 22273 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:56.061491 22273 heartbeater.cc:511] Master 127.11.106.126:42087 requested a full tablet report, sending...
I20260619 06:58:56.062599 21948 ts_manager.cc:194] Registered new tserver with Master: c24f2f0b2e6f496fa5e753d748d3ae8b (127.11.106.66:45269)
I20260619 06:58:56.063284 21948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:41575
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:58:56.083858 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42087
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:58:56.195684 22278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:58:56.195958 22278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:58:56.196002 22278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:58:56.200181 22278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:58:56.200253 22278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:58:56.200313 22278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:58:56.205875 22278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:58:56.206990 22278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:58:56.208103 22278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:58:56.220407 22283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:56.220407 22284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:58:56.220423 22286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:58:56.220614 22278 server_base.cc:1034] running on GCE node
I20260619 06:58:56.221192 22278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:58:56.221797 22278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:58:56.223001 22278 hybrid_clock.cc:648] HybridClock initialized: now 1781852336222992 us; error 39 us; skew 500 ppm
Jun 19 06:58:56 dist-test-slave-10d0 krb5kdc[21903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852336, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:58:56.225947 22278 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:58:56.226821 22278 webserver.cc:460] Webserver started at http://127.11.106.67:45271/ using document root <none> and password file <none>
I20260619 06:58:56.227396 22278 fs_manager.cc:359] Metadata directory not provided
I20260619 06:58:56.227487 22278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:58:56.227726 22278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:58:56.229816 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689"
format_stamp: "Formatted at 2026-06-19 06:58:56 on dist-test-slave-10d0"
server_key: "b6a576cadd77affce14677be84830a3b"
server_key_iv: "b196892ed122dabeeeb4773fca9e936b"
server_key_version: "encryptionkey@0"
I20260619 06:58:56.230383 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689"
format_stamp: "Formatted at 2026-06-19 06:58:56 on dist-test-slave-10d0"
server_key: "b6a576cadd77affce14677be84830a3b"
server_key_iv: "b196892ed122dabeeeb4773fca9e936b"
server_key_version: "encryptionkey@0"
I20260619 06:58:56.234090 22278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:58:56.236413 22293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:56.237629 22278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:58:56.237763 22278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689"
format_stamp: "Formatted at 2026-06-19 06:58:56 on dist-test-slave-10d0"
server_key: "b6a576cadd77affce14677be84830a3b"
server_key_iv: "b196892ed122dabeeeb4773fca9e936b"
server_key_version: "encryptionkey@0"
I20260619 06:58:56.237872 22278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:58:56.276239 22278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:58:56.279788 22278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:58:56.279989 22278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:58:56.280654 22278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:58:56.281508 22278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:58:56.281582 22278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:56.281679 22278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:58:56.281725 22278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:58:56.290997 22278 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:45379
I20260619 06:58:56.291023 22406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:45379 every 8 connection(s)
I20260619 06:58:56.291898 22278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:58:56.301100 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 22278
I20260619 06:58:56.301231 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:58:56.301517 11689 external_mini_cluster.cc:1428] Setting key 9c8f5ce0f75d85d6cb6c5d94aea92011
Jun 19 06:58:56 dist-test-slave-10d0 krb5kdc[21903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852336, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:58:56.306037 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.293857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:33121 (local address 127.11.106.126:42087)
0619 06:58:56.294015 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:56.294019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:56.294894 (+ 875us) server_negotiation.cc:407] Connection header received
0619 06:58:56.296300 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:56.296304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:56.296365 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:56.296469 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:56.297632 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.298163 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.299309 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.299532 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.299645 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.302455 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:56.302460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:56.302462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:56.302485 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:56.304153 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:56.304698 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:56.304702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:56.304703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:56.304782 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:56.305094 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:56.305096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:56.305097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:56.305269 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:56.305344 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:56.305820 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 06:58:56.305917 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:58:56.306895 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.294269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42087 (local address 127.11.106.67:33121)
0619 06:58:56.294742 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:56.294785 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:56.296070 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:56.296667 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:56.296674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:56.297026 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:58:56.297481 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.297494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.298298 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.298318 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.299179 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.299187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.299694 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.299698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.299857 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.300528 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:58:56.300591 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:58:56.302283 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:58:56.304252 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:56.304257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:56.304269 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:56.304509 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:56.304865 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:58:56.304868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:58:56.304869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:58:56.305005 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:58:56.305354 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:58:56.305359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:58:56.305580 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 06:58:56.305755 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 06:58:56.306005 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":98,"threads_started":1}
I20260619 06:58:56.308075 22407 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42087
I20260619 06:58:56.308344 22407 heartbeater.cc:464] Registering TS with master...
I20260619 06:58:56.308895 22407 heartbeater.cc:511] Master 127.11.106.126:42087 requested a full tablet report, sending...
I20260619 06:58:56.309899 21948 ts_manager.cc:194] Registered new tserver with Master: e3cf56c7d3f74f4eb66ff7bf254c0689 (127.11.106.67:45379)
I20260619 06:58:56.310428 21948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:33121
I20260619 06:58:56.315569 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:58:56.325388 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.317527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.11.106.126:42087)
0619 06:58:56.317665 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:56.317670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:56.318035 (+ 365us) server_negotiation.cc:407] Connection header received
0619 06:58:56.318136 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:56.318141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:56.318211 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:56.318316 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:58:56.319213 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.319725 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.320498 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.320763 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.320841 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.322253 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:58:56.322256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:58:56.322258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:58:56.322282 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:58:56.323840 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:56.324253 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:56.324255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:56.324256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:56.324328 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:58:56.324617 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:58:56.324619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:58:56.324620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:58:56.324837 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:58:56.324944 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:56.325177 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 06:58:56.325268 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:58:56.328337 21948 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:58:56.331400 21948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:58:56.346180 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.340278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:45269 (local address 127.0.0.1:54192)
0619 06:58:56.341351 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:56.341364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:56.341571 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:56.342310 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:56.342312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:56.342341 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:56.342447 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.342453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.343417 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.343423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.344677 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.344693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.345618 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.345621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.345726 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.345765 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:58:56.345881 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 06:58:56.345994 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":6,"spinlock_wait_cycles":7424,"thread_start_us":111,"threads_started":1}
I20260619 06:58:56.346199 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.340419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.11.106.66:45269)
0619 06:58:56.340893 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:56.340905 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:56.341455 (+ 550us) server_negotiation.cc:407] Connection header received
0619 06:58:56.341764 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:56.341773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:56.341958 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:56.342107 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:56.342573 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.343290 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.344799 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.345497 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.345606 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.345751 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:56.345924 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 06:58:56.346028 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":141,"threads_started":1}
I20260619 06:58:56.346678 22418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.340106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:45379 (local address 127.0.0.1:53570)
0619 06:58:56.341047 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:56.341096 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:56.341713 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:56.342308 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:56.342311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:56.342335 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:56.342440 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.342448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.343806 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.343810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.345048 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.345058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.346277 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.346282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.346372 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.346401 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:58:56.346475 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:58:56.346544 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"spinlock_wait_cycles":1920,"thread_start_us":68,"threads_started":1}
I20260619 06:58:56.347010 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.341203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.11.106.65:43873)
0619 06:58:56.341585 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:56.341592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:56.341618 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:58:56.341807 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:56.341813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:56.342009 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:56.342159 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:56.342464 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.343790 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.345342 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.346071 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.346167 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.346316 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:56.346768 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 06:58:56.346871 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":97,"threads_started":1}
I20260619 06:58:56.347033 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.340291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.11.106.67:45379)
0619 06:58:56.340802 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:56.340814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:56.341620 (+ 806us) server_negotiation.cc:407] Connection header received
0619 06:58:56.341845 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:56.341849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:56.342009 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:56.342151 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:56.342575 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.343659 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.345172 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.345983 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.346080 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.346207 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:56.346518 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 06:58:56.346604 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":148,"threads_started":1}
I20260619 06:58:56.347251 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.341048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:43873 (local address 127.0.0.1:41816)
0619 06:58:56.341456 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:56.341470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:56.341569 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:56.342172 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:56.342177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:56.342210 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:56.342341 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.342348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.344115 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.344118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.345229 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.345235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.346386 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.346390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.346471 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.346492 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:58:56.346556 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 06:58:56.346628 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":91,"threads_started":1}
I20260619 06:58:56.349857 22073 tablet_service.cc:1467] Processing CreateTablet for tablet d36df1b972bc4fb892cdf0203673f06c (DEFAULT_TABLE table=test-table [id=a9991c6a198448a0b3548dcd60a8456d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:56.349872 22207 tablet_service.cc:1467] Processing CreateTablet for tablet d36df1b972bc4fb892cdf0203673f06c (DEFAULT_TABLE table=test-table [id=a9991c6a198448a0b3548dcd60a8456d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:56.350309 22341 tablet_service.cc:1467] Processing CreateTablet for tablet d36df1b972bc4fb892cdf0203673f06c (DEFAULT_TABLE table=test-table [id=a9991c6a198448a0b3548dcd60a8456d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:58:56.350874 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36df1b972bc4fb892cdf0203673f06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:56.350891 22207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36df1b972bc4fb892cdf0203673f06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:56.351171 22341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36df1b972bc4fb892cdf0203673f06c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:58:56.356937 22424 tablet_bootstrap.cc:492] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54: Bootstrap starting.
I20260619 06:58:56.358356 22426 tablet_bootstrap.cc:492] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: Bootstrap starting.
I20260619 06:58:56.359254 22424 tablet_bootstrap.cc:654] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:56.359983 22424 log.cc:824] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:56.360452 22426 tablet_bootstrap.cc:654] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:56.360905 22425 tablet_bootstrap.cc:492] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b: Bootstrap starting.
I20260619 06:58:56.361318 22426 log.cc:824] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:56.362207 22424 tablet_bootstrap.cc:492] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54: No bootstrap required, opened a new log
I20260619 06:58:56.362411 22424 ts_tablet_manager.cc:1397] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:58:56.363629 22425 tablet_bootstrap.cc:654] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b: Neither blocks nor log segments found. Creating new log.
I20260619 06:58:56.363720 22426 tablet_bootstrap.cc:492] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: No bootstrap required, opened a new log
I20260619 06:58:56.363907 22426 ts_tablet_manager.cc:1397] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:58:56.364449 22425 log.cc:824] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:58:56.366266 22424 raft_consensus.cc:348] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } }
I20260619 06:58:56.366402 22425 tablet_bootstrap.cc:492] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b: No bootstrap required, opened a new log
I20260619 06:58:56.366508 22424 raft_consensus.cc:374] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:56.366585 22425 ts_tablet_manager.cc:1397] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 06:58:56.366582 22424 raft_consensus.cc:729] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ebbd8f8892645a897253c13d7ec0e54, State: Initialized, Role: FOLLOWER
I20260619 06:58:56.367048 22424 consensus_queue.cc:260] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } }
I20260619 06:58:56.367699 22139 heartbeater.cc:503] Master 127.11.106.126:42087 was elected leader, sending a full tablet report...
I20260619 06:58:56.367746 22426 raft_consensus.cc:348] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } }
I20260619 06:58:56.367980 22426 raft_consensus.cc:374] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:56.368021 22426 raft_consensus.cc:729] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cf56c7d3f74f4eb66ff7bf254c0689, State: Initialized, Role: FOLLOWER
I20260619 06:58:56.368042 22424 ts_tablet_manager.cc:1428] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260619 06:58:56.368916 22426 consensus_queue.cc:260] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } }
I20260619 06:58:56.369891 22426 ts_tablet_manager.cc:1428] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 06:58:56.370301 22407 heartbeater.cc:503] Master 127.11.106.126:42087 was elected leader, sending a full tablet report...
I20260619 06:58:56.370312 22425 raft_consensus.cc:348] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } }
I20260619 06:58:56.370574 22425 raft_consensus.cc:374] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:58:56.370632 22425 raft_consensus.cc:729] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24f2f0b2e6f496fa5e753d748d3ae8b, State: Initialized, Role: FOLLOWER
I20260619 06:58:56.371076 22425 consensus_queue.cc:260] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } }
I20260619 06:58:56.372022 22425 ts_tablet_manager.cc:1428] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 06:58:56.372259 22273 heartbeater.cc:503] Master 127.11.106.126:42087 was elected leader, sending a full tablet report...
I20260619 06:58:56.510999 22431 raft_consensus.cc:482] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:58:56.511206 22431 raft_consensus.cc:504] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } }
I20260619 06:58:56.512559 22431 leader_election.cc:288] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c24f2f0b2e6f496fa5e753d748d3ae8b (127.11.106.66:45269), 8ebbd8f8892645a897253c13d7ec0e54 (127.11.106.65:43873)
I20260619 06:58:56.516583 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.512876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:45269 (local address 127.11.106.67:43237)
0619 06:58:56.513055 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:56.513070 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:56.513197 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:56.513620 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:56.513623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:56.513640 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:56.513780 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.513785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.514509 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.514514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.515395 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.515404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.516048 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.516052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.516157 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.516179 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:58:56.516275 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:58:56.516353 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":98,"spinlock_wait_cycles":23936}
I20260619 06:58:56.516580 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.513073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:37887 (local address 127.11.106.65:43873)
0619 06:58:56.513228 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:56.513233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:56.513377 (+ 144us) server_negotiation.cc:407] Connection header received
0619 06:58:56.513488 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:56.513491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:56.513538 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:56.513615 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:56.513915 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.514401 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.515524 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.515960 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.516087 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.516140 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:56.516352 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 06:58:56.516435 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:58:56.516592 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.512903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:43873 (local address 127.11.106.67:37887)
0619 06:58:56.513283 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:56.513295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:58:56.513404 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:58:56.513635 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:58:56.513638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:58:56.513654 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:58:56.513803 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.513808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.514510 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.514515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.515396 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:58:56.515405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.516087 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.516090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:58:56.516197 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.516218 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:58:56.516336 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 06:58:56.516406 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":16,"thread_start_us":126,"threads_started":1}
I20260619 06:58:56.516728 22421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.513271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:43237 (local address 127.11.106.66:45269)
0619 06:58:56.513378 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:56.513381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:56.513394 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:58:56.513443 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:56.513446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:56.513487 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:56.513591 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:58:56.513890 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.514394 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.515517 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.515922 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.516057 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.516124 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:56.516487 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 06:58:56.516585 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 06:58:56.517508 22227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36df1b972bc4fb892cdf0203673f06c" candidate_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" is_pre_election: true
I20260619 06:58:56.517710 22093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36df1b972bc4fb892cdf0203673f06c" candidate_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebbd8f8892645a897253c13d7ec0e54" is_pre_election: true
I20260619 06:58:56.517882 22227 raft_consensus.cc:2454] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cf56c7d3f74f4eb66ff7bf254c0689 in term 0.
I20260619 06:58:56.518018 22093 raft_consensus.cc:2454] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cf56c7d3f74f4eb66ff7bf254c0689 in term 0.
I20260619 06:58:56.518455 22294 leader_election.cc:302] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8ebbd8f8892645a897253c13d7ec0e54, c24f2f0b2e6f496fa5e753d748d3ae8b, e3cf56c7d3f74f4eb66ff7bf254c0689; no voters:
I20260619 06:58:56.518723 22431 raft_consensus.cc:2781] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:58:56.518807 22431 raft_consensus.cc:482] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:58:56.518850 22431 raft_consensus.cc:3037] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:56.519870 22431 raft_consensus.cc:504] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } }
I20260619 06:58:56.520222 22431 leader_election.cc:288] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [CANDIDATE]: Term 1 election: Requested vote from peers c24f2f0b2e6f496fa5e753d748d3ae8b (127.11.106.66:45269), 8ebbd8f8892645a897253c13d7ec0e54 (127.11.106.65:43873)
I20260619 06:58:56.520581 22227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36df1b972bc4fb892cdf0203673f06c" candidate_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b"
I20260619 06:58:56.520645 22093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36df1b972bc4fb892cdf0203673f06c" candidate_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ebbd8f8892645a897253c13d7ec0e54"
I20260619 06:58:56.520697 22227 raft_consensus.cc:3037] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:56.520780 22093 raft_consensus.cc:3037] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:58:56.521728 22227 raft_consensus.cc:2454] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cf56c7d3f74f4eb66ff7bf254c0689 in term 1.
I20260619 06:58:56.521942 22093 raft_consensus.cc:2454] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cf56c7d3f74f4eb66ff7bf254c0689 in term 1.
I20260619 06:58:56.522012 22294 leader_election.cc:302] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c24f2f0b2e6f496fa5e753d748d3ae8b, e3cf56c7d3f74f4eb66ff7bf254c0689; no voters:
I20260619 06:58:56.522199 22431 raft_consensus.cc:2781] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:58:56.522411 22431 raft_consensus.cc:686] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 1 LEADER]: Becoming Leader. State: Replica: e3cf56c7d3f74f4eb66ff7bf254c0689, State: Running, Role: LEADER
I20260619 06:58:56.522761 22431 consensus_queue.cc:237] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } }
I20260619 06:58:56.525964 21947 catalog_manager.cc:5495] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cf56c7d3f74f4eb66ff7bf254c0689 (127.11.106.67). New cstate: current_term: 1 leader_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cf56c7d3f74f4eb66ff7bf254c0689" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 45379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 } health_report { overall_health: UNKNOWN } } }
W20260619 06:58:56.542869 22408 tablet.cc:2369] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:58:56.544268 22274 tablet.cc:2369] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:58:56.572219 22140 tablet.cc:2369] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:58:56.581357 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.577915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.11.106.67:45379)
0619 06:58:56.578053 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 06:58:56.578056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:58:56.578071 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:58:56.578150 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:58:56.578153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:58:56.578209 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:58:56.578297 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:58:56.578546 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.579089 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.579896 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:58:56.580098 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:58:56.580177 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:58:56.580434 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:58:56.580947 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:58:56.581042 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:58:56.581109 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 06:58:56.581164 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:58:56.587524 22227 raft_consensus.cc:1261] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b [term 1 FOLLOWER]: Refusing update from remote peer e3cf56c7d3f74f4eb66ff7bf254c0689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:56.587524 22093 raft_consensus.cc:1261] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54 [term 1 FOLLOWER]: Refusing update from remote peer e3cf56c7d3f74f4eb66ff7bf254c0689: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:58:56.588361 22431 consensus_queue.cc:1035] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24f2f0b2e6f496fa5e753d748d3ae8b" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:56.588604 22434 consensus_queue.cc:1035] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ebbd8f8892645a897253c13d7ec0e54" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 43873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:58:56.606266 21947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:56.606426 21947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:58:56.609328 21947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cc656e21fc54f27809758cff7ecf557: Sending DeleteTablet for 3 replicas of tablet d36df1b972bc4fb892cdf0203673f06c
I20260619 06:58:56.610167 22341 tablet_service.cc:1514] Processing DeleteTablet for tablet d36df1b972bc4fb892cdf0203673f06c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:56 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:53570
I20260619 06:58:56.610409 22073 tablet_service.cc:1514] Processing DeleteTablet for tablet d36df1b972bc4fb892cdf0203673f06c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:56 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:41816
I20260619 06:58:56.610422 22207 tablet_service.cc:1514] Processing DeleteTablet for tablet d36df1b972bc4fb892cdf0203673f06c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:58:56 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:54192
I20260619 06:58:56.610905 22448 tablet_replica.cc:330] T d36df1b972bc4fb892cdf0203673f06c P 8ebbd8f8892645a897253c13d7ec0e54: stopping tablet replica
I20260619 06:58:56.610906 22447 tablet_replica.cc:330] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: stopping tablet replica
I20260619 06:58:56.611258 22447 raft_consensus.cc:2229] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:58:56.611354 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 22010
I20260619 06:58:56.613517 22447 raft_consensus.cc:2258] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:58:56.614929 22447 ts_tablet_manager.cc:1905] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:58:56.617307 22447 ts_tablet_manager.cc:1918] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:58:56.617425 22447 log.cc:1196] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/d36df1b972bc4fb892cdf0203673f06c
I20260619 06:58:56.617746 22447 ts_tablet_manager.cc:1939] T d36df1b972bc4fb892cdf0203673f06c P e3cf56c7d3f74f4eb66ff7bf254c0689: Deleting consensus metadata
I20260619 06:58:56.618886 21933 catalog_manager.cc:4841] TS e3cf56c7d3f74f4eb66ff7bf254c0689 (127.11.106.67:45379): tablet d36df1b972bc4fb892cdf0203673f06c (table test-table [id=a9991c6a198448a0b3548dcd60a8456d]) successfully deleted
W20260619 06:58:56.619846 21934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:43873 (error 108)
I20260619 06:58:56.620034 22449 tablet_replica.cc:330] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b: stopping tablet replica
I20260619 06:58:56.620429 22449 raft_consensus.cc:2229] T d36df1b972bc4fb892cdf0203673f06c P c24f2f0b2e6f496fa5e753d748d3ae8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:58:56.620745 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 22144
I20260619 06:58:56.621016 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.620722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:43873 (local address 127.0.0.1:41820)
0619 06:58:56.620848 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:56.620919 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:43873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:58:56.621464 21934 catalog_manager.cc:4584] TS 8ebbd8f8892645a897253c13d7ec0e54 (127.11.106.65:43873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d36df1b972bc4fb892cdf0203673f06c: Network error: Client connection negotiation failed: client connection to 127.11.106.65:43873: connect: Connection refused (error 111)
I20260619 06:58:56.628130 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 22278
I20260619 06:58:56.629732 22419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:58:56.628259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:45269 (local address 127.0.0.1:54206)
0619 06:58:56.629509 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0619 06:58:56.629580 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.66:45269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1147}
I20260619 06:58:56.634166 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 21916
2026-06-19T06:58:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2457 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:58:56 dist-test-slave-10d0 krb5kdc[22453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:58:56 dist-test-slave-10d0 krb5kdc[22453](info): set up 2 sockets
Jun 19 06:58:56 dist-test-slave-10d0 krb5kdc[22453](info): commencing operation
krb5kdc: starting...
W20260619 06:58:57.682224 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:58:57 dist-test-slave-10d0 krb5kdc[22453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:58:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.106.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.106.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/rangeradmin_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/rangeradmin_127.11.106.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.106.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.106.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/rangerlookup_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/rangerlookup_127.11.106.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.106.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.106.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/HTTP_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/HTTP_127.11.106.84@KRBTEST.COM.keytab.
I20260619 06:58:57.855132 11689 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1_IlA7/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/postgres -l logfile start
2026-06-19 06:59:00.406 UTC [22483] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:59:00.406 UTC [22483] LOG: listening on IPv4 address "127.11.106.84", port 57025
2026-06-19 06:59:00.410 UTC [22483] LOG: listening on Unix socket "/tmp/.s.PGSQL.57025"
2026-06-19 06:59:00.431 UTC [22486] LOG: database system was shut down at 2026-06-19 06:58:58 UTC
2026-06-19 06:59:00.435 UTC [22483] LOG: database system is ready to accept connections
I20260619 06:59:01.390112 11689 mini_postgres.cc:96] Postgres bound to 57025
2026-06-19 06:59:01.396 UTC [22494] FATAL: database "slave" does not exist
127.11.106.84:57025 - accepting connections
I20260619 06:59:01.397570 11689 mini_ranger.cc:161] Starting Ranger...
I20260619 06:59:01.413327 11689 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:59:01.946650 11689 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:59:01.946764 11689 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-admin
2026-06-19 06:59:02,235 [I] DB FLAVOR :POSTGRES
2026-06-19 06:59:02,235 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:59:02,235 [I] Checking connection..
2026-06-19 06:59:02,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:59:02,532 [I] Checking connection passed.
2026-06-19 06:59:02,533 [I] --------- Verifying version history table ---------
2026-06-19 06:59:02,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:59:02,774 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:59:02,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:59:03,020 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:59:03,020 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:59:03,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:59:03.279 UTC [22558] WARNING: there is no transaction in progress
2026-06-19 06:59:03,286 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:59:03,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:59:03,541 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:59:03,541 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:59:03,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:59:03,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:59:04,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-10d0.c.gcp-upstream.internal','N') ;"
2026-06-19 06:59:04,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:59:04,513 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:59:04,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:59:04,753 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:59:04,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:59:05,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:59:05,250 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:59:05,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:59:06.258 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.274 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.291 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.306 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.322 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.340 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.401 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.408 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.423 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.434 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.444 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.459 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.473 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.480 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.487 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:06.494 UTC [22666] WARNING: there is no transaction in progress
2026-06-19 06:59:07,151 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:59:07,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:59:07,374 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:59:07,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:59:07,640 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:59:07,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:59:07,882 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:59:07,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:59:08,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:59:08,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-10d0.c.gcp-upstream.internal';"
2026-06-19 06:59:08,618 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:59:08,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:59:08,861 [I] DB_PATCHES have already been applied
I20260619 06:59:08.866950 11689 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-admin:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1_IlA7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:59:08.867008 11689 mini_ranger.cc:193] Using host: 127.11.106.84
I20260619 06:59:08.870585 11689 mini_ranger.cc:227] Ranger admin URL: http://127.11.106.84:54815
Jun 19, 2026 6:59:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:59:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:59:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:59:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:59:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:59:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:59:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:59:10.039083 11689 mini_ranger.cc:160] Time spent starting Ranger: real 8.642s user 0.003s sys 0.007s
Jun 19, 2026 6:59:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:59:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:59:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:59:30 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:59:31 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:59:32.248000 11689 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.106.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.106.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/rangerkms_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/rangerkms_127.11.106.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.106.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.106.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.106.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/HTTP_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.106.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/HTTP_127.11.106.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:59:32 dist-test-slave-10d0 krb5kdc[22453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852372, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:59:32.349459 11689 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:59:32.359782 11689 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:59:33.243619 11689 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:59:33.243721 11689 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-kms
I20260619 06:59:33.243825 11689 mini_ranger_kms.cc:227] Using postgres at 127.11.106.84:57025
2026-06-19 06:59:33,496 [I] DB FLAVOR :POSTGRES
2026-06-19 06:59:33,497 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:59:33,497 [I] Checking connection
2026-06-19 06:59:33,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:59:33,741 [I] connection success
2026-06-19 06:59:33,741 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:59:33,741 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:59:33,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:59:34,003 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:59:34,003 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:59:34,003 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:59:34,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:57025/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:59:34,260 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:59:34.767930 11689 mini_ranger_kms.cc:310] Created kms service
I20260619 06:59:34.965329 11689 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:59:35.010177 11689 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:59:35 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852372, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 06:59:35.428719 11689 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:59:35.428877 11689 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-kms:/tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:59:35.428901 11689 mini_ranger_kms.cc:240] Using host: 127.11.106.84
I20260619 06:59:35.433027 11689 mini_ranger_kms.cc:276] Ranger KMS PID: 22846
I20260619 06:59:35.433161 11689 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.106.84:34273
Jun 19, 2026 6:59:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:59:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:59:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:59:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:59:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:59:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 06:59:37.681303 11689 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.332s user 0.002s sys 0.232s
I20260619 06:59:37.681420 11689 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:59:37.681516 11689 mini_ranger_kms.cc:401] 127.11.106.84:34273/kms/v1/keys
Jun 19 06:59:37 dist-test-slave-10d0 krb5kdc[22453](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852377, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.106.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 06:59:38 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852377, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.106.84@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
Jun 19 06:59:38 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852377, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.106.84@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
Jun 19 06:59:40 dist-test-slave-10d0 krb5kdc[22453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 06:59:40.971586 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36185
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:36185
--ranger_config_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:34273/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:59:41.084357 22897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:59:41.084709 22897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:59:41.084793 22897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:59:41.089180 22897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:59:41.089275 22897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:59:41.089314 22897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:59:41.089349 22897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:59:41.089385 22897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:59:41.094897 22897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.106.84:34273/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:36185
--ranger_config_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:59:41.096050 22897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:59:41.097126 22897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:59:41.105386 22902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:59:41.105365 22903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:59:41.105556 22905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:59:41.105978 22897 server_base.cc:1034] running on GCE node
I20260619 06:59:41.106519 22897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:59:41.107666 22897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:59:41.108850 22897 hybrid_clock.cc:648] HybridClock initialized: now 1781852381108834 us; error 33 us; skew 500 ppm
Jun 19 06:59:41 dist-test-slave-10d0 krb5kdc[22453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852381, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:59:41.111774 22897 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 06:59:41.112811 22897 webserver.cc:460] Webserver started at http://127.11.106.126:34601/ using document root <none> and password file <none>
I20260619 06:59:41.113334 22897 fs_manager.cc:359] Metadata directory not provided
I20260619 06:59:41.113386 22897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:59:41.113547 22897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:59:41 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852381, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 06:59:41.168735 22897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "57a0ade9a7364568b163684d39e4b4b3"
format_stamp: "Formatted at 2026-06-19 06:59:41 on dist-test-slave-10d0"
server_key: "7cda648059f96089ef122ae44132b13b"
server_key_iv: "b41a403d589cec9e8dbf64744fec4ae3"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:41.169374 22897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "57a0ade9a7364568b163684d39e4b4b3"
format_stamp: "Formatted at 2026-06-19 06:59:41 on dist-test-slave-10d0"
server_key: "7cda648059f96089ef122ae44132b13b"
server_key_iv: "b41a403d589cec9e8dbf64744fec4ae3"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:41.175195 22897 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 06:59:41.191162 22912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:41.193001 22897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:59:41.193135 22897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "57a0ade9a7364568b163684d39e4b4b3"
format_stamp: "Formatted at 2026-06-19 06:59:41 on dist-test-slave-10d0"
server_key: "7cda648059f96089ef122ae44132b13b"
server_key_iv: "b41a403d589cec9e8dbf64744fec4ae3"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:41.193288 22897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:59:41.210630 22897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:59:41.214136 22897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:59:41.214357 22897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:59:41.222731 22897 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:36185
I20260619 06:59:41.222813 22964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:36185 every 8 connection(s)
I20260619 06:59:41.223830 22897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 06:59:41.226815 22965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:59:41.229351 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 22897
I20260619 06:59:41.229480 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 06:59:41.229822 11689 external_mini_cluster.cc:1428] Setting key 56f04eaa73d34aa3c53800ce6b189b11
I20260619 06:59:41.233855 22965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3: Bootstrap starting.
I20260619 06:59:41.236503 22965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3: Neither blocks nor log segments found. Creating new log.
I20260619 06:59:41.237468 22965 log.cc:824] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:59:41 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:59:41.239768 22965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3: No bootstrap required, opened a new log
I20260619 06:59:41.244431 22965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a0ade9a7364568b163684d39e4b4b3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36185 } }
I20260619 06:59:41.244727 22965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:59:41.244786 22965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a0ade9a7364568b163684d39e4b4b3, State: Initialized, Role: FOLLOWER
I20260619 06:59:41.245271 22965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a0ade9a7364568b163684d39e4b4b3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36185 } }
I20260619 06:59:41.245421 22965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:59:41.245447 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:41.231335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.11.106.126:36185)
0619 06:59:41.231763 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:41.231784 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:41.231830 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:59:41.232722 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:41.232747 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:41.233191 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:41.233552 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:59:41.234335 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:41.235805 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:41.236801 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:41.237505 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:41.237662 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:41.240424 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:59:41.240433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:59:41.240453 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:59:41.240489 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:59:41.242318 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:41.242748 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:41.242754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:41.242759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:41.242888 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:41.243201 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:41.243204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:41.243206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:41.243720 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:59:41.243922 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:41.244186 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 06:59:41.244459 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260619 06:59:41.245496 22965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:59:41.245585 22965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:59:41.246512 22965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a0ade9a7364568b163684d39e4b4b3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36185 } }
I20260619 06:59:41.246888 22965 leader_election.cc:302] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57a0ade9a7364568b163684d39e4b4b3; no voters:
I20260619 06:59:41.247200 22965 leader_election.cc:288] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:59:41.247308 22970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:59:41.247517 22970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [term 1 LEADER]: Becoming Leader. State: Replica: 57a0ade9a7364568b163684d39e4b4b3, State: Running, Role: LEADER
I20260619 06:59:41.247886 22970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a0ade9a7364568b163684d39e4b4b3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36185 } }
I20260619 06:59:41.248428 22965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:59:41.249768 22971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57a0ade9a7364568b163684d39e4b4b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a0ade9a7364568b163684d39e4b4b3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36185 } } }
I20260619 06:59:41.249893 22971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:59:41.250859 22965 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:59:41.250928 22972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57a0ade9a7364568b163684d39e4b4b3. Latest consensus state: current_term: 1 leader_uuid: "57a0ade9a7364568b163684d39e4b4b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a0ade9a7364568b163684d39e4b4b3" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 36185 } } }
I20260619 06:59:41.251022 22972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:59:41.251346 22980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:59:41.257158 22980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:59:41.264174 22980 catalog_manager.cc:1347] Generated new cluster ID: 96756eddc82c47acb5491b766e29bb80
I20260619 06:59:41.264289 22980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:59:41.278234 22980 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:59:41.279268 22980 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:59:41.288971 22980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3: Generated new TSK 0
I20260619 06:59:41.289680 22980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 06:59:42.008780 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:34273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36185
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:59:42.171213 23010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:59:42.171514 23010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:59:42.171627 23010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:59:42.175730 23010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:59:42.175818 23010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:59:42.175910 23010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 06:59:42.182267 23010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.106.84:34273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:59:42.183420 23010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:59:42.184495 23010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:59:42.194073 23020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:59:42.194093 23017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:59:42.194521 23010 server_base.cc:1034] running on GCE node
W20260619 06:59:42.194764 23018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:59:42.195194 23010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:59:42.195894 23010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:59:42.197083 23010 hybrid_clock.cc:648] HybridClock initialized: now 1781852382197049 us; error 45 us; skew 500 ppm
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:59:42.200062 23010 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 06:59:42.201122 23010 webserver.cc:460] Webserver started at http://127.11.106.65:36043/ using document root <none> and password file <none>
I20260619 06:59:42.201700 23010 fs_manager.cc:359] Metadata directory not provided
I20260619 06:59:42.201762 23010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:59:42.201978 23010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 06:59:42.223367 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "afd46dd4054746b38ddf9ea855eafbdb"
format_stamp: "Formatted at 2026-06-19 06:59:42 on dist-test-slave-10d0"
server_key: "f8581ff0eb6efbdf8013e13ac0b5afd7"
server_key_iv: "5148db603c6ad1b00aabd7278052fd93"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:42.224139 23010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "afd46dd4054746b38ddf9ea855eafbdb"
format_stamp: "Formatted at 2026-06-19 06:59:42 on dist-test-slave-10d0"
server_key: "f8581ff0eb6efbdf8013e13ac0b5afd7"
server_key_iv: "5148db603c6ad1b00aabd7278052fd93"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:42.230298 23010 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 06:59:42.269126 23027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:42.271261 23010 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.000s
I20260619 06:59:42.271421 23010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "afd46dd4054746b38ddf9ea855eafbdb"
format_stamp: "Formatted at 2026-06-19 06:59:42 on dist-test-slave-10d0"
server_key: "f8581ff0eb6efbdf8013e13ac0b5afd7"
server_key_iv: "5148db603c6ad1b00aabd7278052fd93"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:42.271587 23010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:59:42.291612 23010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:59:42.295375 23010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:59:42.295645 23010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:59:42.296380 23010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:59:42.297552 23010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:59:42.297658 23010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:42.297775 23010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:59:42.297852 23010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:42.308574 23010 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:45545
I20260619 06:59:42.308739 23142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:45545 every 8 connection(s)
I20260619 06:59:42.309878 23010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 06:59:42.312069 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 23010
I20260619 06:59:42.312242 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 06:59:42.312598 11689 external_mini_cluster.cc:1428] Setting key d27235dac144d1f5aa39cb10ea9f85fd
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 06:59:42.329957 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:42.313238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:59917 (local address 127.11.106.126:36185)
0619 06:59:42.314664 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:42.314669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:42.314685 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:59:42.316989 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:42.316993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:42.317052 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:42.317123 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:59:42.318956 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:42.319848 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:42.321630 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:42.321989 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:42.322080 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:42.325229 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:59:42.325235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:59:42.325239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:59:42.325270 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:59:42.327212 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:42.327910 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:42.327914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:42.327916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:42.328016 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:42.328473 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:42.328476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:42.328478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:42.328762 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:59:42.328855 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:42.329618 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 06:59:42.329753 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":61,"threads_started":1}
I20260619 06:59:42.331646 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:42.312682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36185 (local address 127.11.106.65:59917)
0619 06:59:42.313450 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 06:59:42.313511 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:59:42.316708 (+ 3197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:59:42.317606 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:59:42.317618 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:59:42.318002 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:59:42.318769 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:42.318791 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:42.320024 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:42.320027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:42.321473 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:42.321485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:42.322325 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:42.322329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:42.322557 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:42.323282 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:59:42.323304 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:59:42.325009 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:59:42.327352 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:59:42.327359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:59:42.327378 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:59:42.327779 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:59:42.328138 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:59:42.328141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:59:42.328146 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:59:42.328345 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:59:42.328930 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:59:42.328937 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:59:42.329293 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 06:59:42.330152 (+ 859us) client_negotiation.cc:240] Negotiation successful
0619 06:59:42.330548 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":141,"threads_started":1}
I20260619 06:59:42.333433 23143 heartbeater.cc:347] Connected to a master server at 127.11.106.126:36185
I20260619 06:59:42.333707 23143 heartbeater.cc:464] Registering TS with master...
I20260619 06:59:42.334299 23143 heartbeater.cc:511] Master 127.11.106.126:36185 requested a full tablet report, sending...
I20260619 06:59:42.336129 22923 ts_manager.cc:194] Registered new tserver with Master: afd46dd4054746b38ddf9ea855eafbdb (127.11.106.65:45545)
I20260619 06:59:42.337725 22923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:59917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 06:59:42.348354 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:34273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36185
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:59:42.507092 23149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:59:42.507500 23149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:59:42.507602 23149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:59:42.513793 23149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:59:42.513890 23149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:59:42.513993 23149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 06:59:42.522364 23149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.106.84:34273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:59:42.523849 23149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:59:42.525331 23149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:59:42.537940 23156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:59:42.538033 23149 server_base.cc:1034] running on GCE node
W20260619 06:59:42.538288 23158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:59:42.538122 23155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:59:42.538956 23149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:59:42.539791 23149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:59:42.541000 23149 hybrid_clock.cc:648] HybridClock initialized: now 1781852382540973 us; error 44 us; skew 500 ppm
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:59:42.544813 23149 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 06:59:42.545989 23149 webserver.cc:460] Webserver started at http://127.11.106.66:35933/ using document root <none> and password file <none>
I20260619 06:59:42.546802 23149 fs_manager.cc:359] Metadata directory not provided
I20260619 06:59:42.546885 23149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:59:42.547290 23149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 06:59:42.579234 23149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "c30be41cfd97464d9fb576893ce9a827"
format_stamp: "Formatted at 2026-06-19 06:59:42 on dist-test-slave-10d0"
server_key: "887d23c5d56a8384d601aece179f2b8d"
server_key_iv: "41fcd934d32cbdb8a2a5a12eb04bb17e"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:42.580044 23149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "c30be41cfd97464d9fb576893ce9a827"
format_stamp: "Formatted at 2026-06-19 06:59:42 on dist-test-slave-10d0"
server_key: "887d23c5d56a8384d601aece179f2b8d"
server_key_iv: "41fcd934d32cbdb8a2a5a12eb04bb17e"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:42.589231 23149 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260619 06:59:42.606326 23165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:42.608606 23149 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:59:42.608819 23149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "c30be41cfd97464d9fb576893ce9a827"
format_stamp: "Formatted at 2026-06-19 06:59:42 on dist-test-slave-10d0"
server_key: "887d23c5d56a8384d601aece179f2b8d"
server_key_iv: "41fcd934d32cbdb8a2a5a12eb04bb17e"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:42.609002 23149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:59:42.634392 23149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:59:42.639564 23149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:59:42.639772 23149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:59:42.640515 23149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:59:42.641644 23149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:59:42.641697 23149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:42.641753 23149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:59:42.641777 23149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:42.653390 23149 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:44349
I20260619 06:59:42.654673 23149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 06:59:42.664475 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 23149
I20260619 06:59:42.664680 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 06:59:42.665040 11689 external_mini_cluster.cc:1428] Setting key a25709efff40a9aefc2b84e43db501a7
I20260619 06:59:42.668967 23278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:44349 every 8 connection(s)
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 06:59:42.694826 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:42.662456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:43035 (local address 127.11.106.126:36185)
0619 06:59:42.662597 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:42.662602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:42.669962 (+ 7360us) server_negotiation.cc:407] Connection header received
0619 06:59:42.672626 (+ 2664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:42.672630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:42.672693 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:42.672790 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:59:42.680406 (+ 7616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:42.681324 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:42.683377 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:42.683753 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:42.683853 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:42.689093 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:59:42.689098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:59:42.689101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:59:42.689137 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:59:42.691154 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:42.691924 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:42.691929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:42.691931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:42.692160 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:42.693127 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:42.693131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:42.693132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:42.693389 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:59:42.693480 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:42.694493 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0619 06:59:42.694650 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 06:59:42.699371 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:42.663112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36185 (local address 127.11.106.66:43035)
0619 06:59:42.669763 (+ 6651us) negotiation.cc:107] Waiting for socket to connect
0619 06:59:42.669830 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 06:59:42.672260 (+ 2430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:59:42.674916 (+ 2656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:59:42.674928 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:59:42.675310 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:59:42.676142 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:42.676163 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:42.681554 (+ 5391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:42.681558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:42.683186 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:42.683202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:42.684131 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:42.684136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:42.684380 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:42.685254 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:59:42.685280 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:59:42.688808 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:59:42.691338 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:59:42.691345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:59:42.691366 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:59:42.691778 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:59:42.692747 (+ 969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:59:42.692751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:59:42.692753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:59:42.692967 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:59:42.693557 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:59:42.693564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:59:42.693978 (+ 414us) client_negotiation.cc:769] Sending connection context
0619 06:59:42.695487 (+ 1509us) client_negotiation.cc:240] Negotiation successful
0619 06:59:42.698146 (+ 2659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6356,"thread_start_us":161,"threads_started":1}
I20260619 06:59:42.701125 23279 heartbeater.cc:347] Connected to a master server at 127.11.106.126:36185
I20260619 06:59:42.701382 23279 heartbeater.cc:464] Registering TS with master...
I20260619 06:59:42.701996 23279 heartbeater.cc:511] Master 127.11.106.126:36185 requested a full tablet report, sending...
I20260619 06:59:42.703303 22923 ts_manager.cc:194] Registered new tserver with Master: c30be41cfd97464d9fb576893ce9a827 (127.11.106.66:44349)
I20260619 06:59:42.704075 22923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:43035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 06:59:42.725878 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:34273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36185
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:39673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
W20260619 06:59:42.915107 23284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:59:42.915591 23284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:59:42.915678 23284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:59:42.921949 23284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:59:42.922036 23284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:59:42.922123 23284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 06:59:42.929901 23284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:39673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.106.84:34273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 06:59:42.931427 23284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:59:42.932859 23284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:59:42.945459 23290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:59:42.946010 23292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:59:42.946228 23289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:59:42.947028 23284 server_base.cc:1034] running on GCE node
I20260619 06:59:42.947592 23284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:59:42.948400 23284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:59:42.963851 23284 hybrid_clock.cc:648] HybridClock initialized: now 1781852382963736 us; error 135 us; skew 500 ppm
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:59:42.967883 23284 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 06:59:42.969216 23284 webserver.cc:460] Webserver started at http://127.11.106.67:36143/ using document root <none> and password file <none>
I20260619 06:59:42.969954 23284 fs_manager.cc:359] Metadata directory not provided
I20260619 06:59:42.970037 23284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:59:42.970273 23284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:59:42 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 06:59:42.985734 23284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "8c38388f77ce4c54a93177fdc101a02d"
format_stamp: "Formatted at 2026-06-19 06:59:42 on dist-test-slave-10d0"
server_key: "ea0b1ae98bc9745b9bd976434f914933"
server_key_iv: "143109b73964c8c607431e9bdaa69b30"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:42.986420 23284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "8c38388f77ce4c54a93177fdc101a02d"
format_stamp: "Formatted at 2026-06-19 06:59:42 on dist-test-slave-10d0"
server_key: "ea0b1ae98bc9745b9bd976434f914933"
server_key_iv: "143109b73964c8c607431e9bdaa69b30"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:43.010407 23284 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.001s sys 0.004s
I20260619 06:59:43.030300 23299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:43.032008 23284 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:59:43.032133 23284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "8c38388f77ce4c54a93177fdc101a02d"
format_stamp: "Formatted at 2026-06-19 06:59:42 on dist-test-slave-10d0"
server_key: "ea0b1ae98bc9745b9bd976434f914933"
server_key_iv: "143109b73964c8c607431e9bdaa69b30"
server_key_version: "kuduclusterkey@0"
I20260619 06:59:43.032270 23284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:59:43.077425 23284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:59:43.081640 23284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:59:43.081902 23284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:59:43.082634 23284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:59:43.083717 23284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:59:43.083818 23284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:43.083912 23284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:59:43.083973 23284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:59:43.097995 23284 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:43499
I20260619 06:59:43.099416 23284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 06:59:43.101267 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 23284
I20260619 06:59:43.101400 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 06:59:43.101739 11689 external_mini_cluster.cc:1428] Setting key c02130c3a1e35e71b1f35c6965bb6319
I20260619 06:59:43.100745 23412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:43499 every 8 connection(s)
Jun 19 06:59:43 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 06:59:43.139665 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.106319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:44267 (local address 127.11.106.126:36185)
0619 06:59:43.106433 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:43.106438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:43.114918 (+ 8480us) server_negotiation.cc:407] Connection header received
0619 06:59:43.116912 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:43.116916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:43.116971 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:43.117022 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:59:43.118874 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.119691 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.121634 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.121962 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.122039 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.128989 (+ 6950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:59:43.128994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:59:43.128996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:59:43.129027 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:59:43.130806 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:43.137292 (+ 6486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:43.137297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:43.137298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:43.137398 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:43.137902 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:43.137907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:43.137908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:43.138157 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:59:43.138247 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:43.139334 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0619 06:59:43.139489 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:59:43.142097 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.105242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:36185 (local address 127.11.106.67:44267)
0619 06:59:43.114712 (+ 9470us) negotiation.cc:107] Waiting for socket to connect
0619 06:59:43.114796 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 06:59:43.116383 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:59:43.117488 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:59:43.117498 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:59:43.117912 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:59:43.118691 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.118712 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.119856 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.119860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.121473 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.121485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.122258 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.122261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.122475 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.126498 (+ 4023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:59:43.126523 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:59:43.128774 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:59:43.136650 (+ 7876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:59:43.136663 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:59:43.136691 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:59:43.137138 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:59:43.137516 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:59:43.137519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:59:43.137521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:59:43.137768 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:59:43.138639 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:59:43.138648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:59:43.139020 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 06:59:43.140354 (+ 1334us) client_negotiation.cc:240] Negotiation successful
0619 06:59:43.140774 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9155,"thread_start_us":157,"threads_started":1}
I20260619 06:59:43.143703 23413 heartbeater.cc:347] Connected to a master server at 127.11.106.126:36185
I20260619 06:59:43.143950 23413 heartbeater.cc:464] Registering TS with master...
I20260619 06:59:43.144527 23413 heartbeater.cc:511] Master 127.11.106.126:36185 requested a full tablet report, sending...
I20260619 06:59:43.145860 22923 ts_manager.cc:194] Registered new tserver with Master: 8c38388f77ce4c54a93177fdc101a02d (127.11.106.67:43499)
I20260619 06:59:43.146626 22923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:44267
I20260619 06:59:43.152772 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:59:43.183351 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.161098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.11.106.126:36185)
0619 06:59:43.161242 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:43.161247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:43.173062 (+ 11815us) server_negotiation.cc:407] Connection header received
0619 06:59:43.173345 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:43.173349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:43.173424 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:43.173483 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:59:43.174598 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.175369 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.176625 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.176960 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.177082 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.179389 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:59:43.179394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:59:43.179400 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:59:43.179438 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:59:43.181258 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:43.181821 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:43.181825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:43.181827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:43.181924 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:59:43.182268 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:59:43.182272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:59:43.182274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:59:43.182511 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:59:43.182624 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:43.183025 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 06:59:43.183166 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:59:43.188055 22923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:59:43.193217 22923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:59:43.222414 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.210457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43499 (local address 127.0.0.1:50596)
0619 06:59:43.212938 (+ 2481us) negotiation.cc:107] Waiting for socket to connect
0619 06:59:43.212952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:59:43.213040 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:59:43.214279 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:59:43.214286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:59:43.214310 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:59:43.214453 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.214460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.217966 (+ 3506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.217970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.220330 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.220343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.221770 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.221774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.221905 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.221951 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 06:59:43.222083 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 06:59:43.222228 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2422,"thread_start_us":112,"threads_started":1}
I20260619 06:59:43.223412 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.210903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.11.106.67:43499)
0619 06:59:43.213550 (+ 2647us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:43.213563 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:43.213589 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:59:43.213684 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:43.213690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:43.213925 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:43.214088 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:59:43.216014 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.217831 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.220507 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.221410 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.221522 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.221701 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:43.223146 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0619 06:59:43.223265 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2533,"thread_start_us":130,"threads_started":1}
I20260619 06:59:43.225329 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.207849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:44349 (local address 127.0.0.1:45700)
0619 06:59:43.212394 (+ 4545us) negotiation.cc:107] Waiting for socket to connect
0619 06:59:43.212455 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:59:43.212637 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:59:43.220628 (+ 7991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:59:43.220633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:59:43.220656 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:59:43.220779 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.220788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.222276 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.222280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.223455 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.223466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.224867 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.224871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.224973 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.225001 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:59:43.225078 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:59:43.225150 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4306,"thread_start_us":109,"threads_started":1}
I20260619 06:59:43.226794 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.208798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.11.106.66:44349)
0619 06:59:43.214769 (+ 5971us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:43.214789 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:43.214815 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:59:43.214891 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:43.214896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:43.215111 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:43.216694 (+ 1583us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:59:43.220950 (+ 4256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.222118 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.223610 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.224497 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.224638 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.224798 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:43.226539 (+ 1741us) server_negotiation.cc:299] Negotiation successful
0619 06:59:43.226652 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5840,"thread_start_us":117,"threads_started":1}
I20260619 06:59:43.227344 23346 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa877fb391f403f8e31a55a731a6f62 (DEFAULT_TABLE table=test-table [id=4441e660107b435db7e8a4f3f59f9089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:59:43.231684 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.211646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45545 (local address 127.0.0.1:36868)
0619 06:59:43.213213 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0619 06:59:43.213225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:59:43.213294 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:59:43.227073 (+ 13779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:59:43.227077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:59:43.227102 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:59:43.227222 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.227229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.228957 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.228960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.230165 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.230174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.231289 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.231293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.231412 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.231439 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:59:43.231486 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 06:59:43.231556 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1509,"thread_start_us":95,"threads_started":1}
I20260619 06:59:43.231750 23213 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa877fb391f403f8e31a55a731a6f62 (DEFAULT_TABLE table=test-table [id=4441e660107b435db7e8a4f3f59f9089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:59:43.232491 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.211837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.11.106.65:45545)
0619 06:59:43.225959 (+ 14122us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:43.225967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:43.225995 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:59:43.226081 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:43.226086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:43.226358 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:43.227398 (+ 1040us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:59:43.227562 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.228832 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.230329 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.231139 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.232057 (+ 918us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.232222 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:43.232272 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 06:59:43.232360 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13995,"thread_start_us":104,"threads_started":1}
I20260619 06:59:43.235854 23065 tablet_service.cc:1467] Processing CreateTablet for tablet 5fa877fb391f403f8e31a55a731a6f62 (DEFAULT_TABLE table=test-table [id=4441e660107b435db7e8a4f3f59f9089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:59:43.237040 23065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa877fb391f403f8e31a55a731a6f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:59:43.239406 23346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa877fb391f403f8e31a55a731a6f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:59:43.239809 23213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fa877fb391f403f8e31a55a731a6f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:59:43.270800 23432 tablet_bootstrap.cc:492] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: Bootstrap starting.
I20260619 06:59:43.272927 23433 tablet_bootstrap.cc:492] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb: Bootstrap starting.
I20260619 06:59:43.274137 23432 tablet_bootstrap.cc:654] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: Neither blocks nor log segments found. Creating new log.
I20260619 06:59:43.274608 23434 tablet_bootstrap.cc:492] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: Bootstrap starting.
I20260619 06:59:43.279736 23432 log.cc:824] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: Log is configured to *not* fsync() on all Append() calls
I20260619 06:59:43.281558 23433 tablet_bootstrap.cc:654] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb: Neither blocks nor log segments found. Creating new log.
I20260619 06:59:43.283414 23434 tablet_bootstrap.cc:654] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: Neither blocks nor log segments found. Creating new log.
I20260619 06:59:43.284422 23433 log.cc:824] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb: Log is configured to *not* fsync() on all Append() calls
I20260619 06:59:43.285349 23434 log.cc:824] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:59:43.288785 23432 tablet_bootstrap.cc:492] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: No bootstrap required, opened a new log
I20260619 06:59:43.289073 23432 ts_tablet_manager.cc:1397] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260619 06:59:43.295049 23432 raft_consensus.cc:348] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } }
I20260619 06:59:43.295578 23432 raft_consensus.cc:374] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:59:43.295710 23432 raft_consensus.cc:729] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30be41cfd97464d9fb576893ce9a827, State: Initialized, Role: FOLLOWER
I20260619 06:59:43.296346 23432 consensus_queue.cc:260] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } }
I20260619 06:59:43.297952 23432 ts_tablet_manager.cc:1428] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:59:43.298451 23279 heartbeater.cc:503] Master 127.11.106.126:36185 was elected leader, sending a full tablet report...
I20260619 06:59:43.304092 23433 tablet_bootstrap.cc:492] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb: No bootstrap required, opened a new log
I20260619 06:59:43.305723 23434 tablet_bootstrap.cc:492] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: No bootstrap required, opened a new log
I20260619 06:59:43.306304 23433 ts_tablet_manager.cc:1397] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.000s
I20260619 06:59:43.314199 23434 ts_tablet_manager.cc:1397] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.000s
Jun 19 06:59:43 dist-test-slave-10d0 krb5kdc[22453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852382, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 06:59:43.326565 23434 raft_consensus.cc:348] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } }
I20260619 06:59:43.327729 23433 raft_consensus.cc:348] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } }
I20260619 06:59:43.328135 23433 raft_consensus.cc:374] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:59:43.328249 23433 raft_consensus.cc:729] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd46dd4054746b38ddf9ea855eafbdb, State: Initialized, Role: FOLLOWER
I20260619 06:59:43.329442 23433 consensus_queue.cc:260] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } }
I20260619 06:59:43.329702 23434 raft_consensus.cc:374] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:59:43.329818 23434 raft_consensus.cc:729] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c38388f77ce4c54a93177fdc101a02d, State: Initialized, Role: FOLLOWER
I20260619 06:59:43.330422 23434 consensus_queue.cc:260] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } }
I20260619 06:59:43.331776 23433 ts_tablet_manager.cc:1428] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260619 06:59:43.332911 23434 ts_tablet_manager.cc:1428] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260619 06:59:43.333398 23143 heartbeater.cc:503] Master 127.11.106.126:36185 was elected leader, sending a full tablet report...
I20260619 06:59:43.334863 23413 heartbeater.cc:503] Master 127.11.106.126:36185 was elected leader, sending a full tablet report...
W20260619 06:59:43.352051 23414 tablet.cc:2369] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:59:43.411046 23280 tablet.cc:2369] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:59:43.442965 23441 raft_consensus.cc:482] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:59:43.443223 23441 raft_consensus.cc:504] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } }
I20260619 06:59:43.444695 23441 leader_election.cc:288] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c30be41cfd97464d9fb576893ce9a827 (127.11.106.66:44349), afd46dd4054746b38ddf9ea855eafbdb (127.11.106.65:45545)
I20260619 06:59:43.451013 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.445154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:44349 (local address 127.11.106.67:43263)
0619 06:59:43.445992 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0619 06:59:43.446008 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:59:43.446108 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:59:43.446920 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:59:43.446923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:59:43.446947 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:59:43.447082 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.447090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.448238 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.448242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.449548 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.449560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.450577 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.450581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.450698 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.450721 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:59:43.450791 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:59:43.450864 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754}
I20260619 06:59:43.451754 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.446392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:43263 (local address 127.11.106.66:44349)
0619 06:59:43.446638 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:43.446643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:43.446659 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:59:43.446726 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:43.446730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:43.446780 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:43.446862 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:59:43.447318 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.448094 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.449707 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.450348 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.450462 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.450516 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:43.451533 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0619 06:59:43.451620 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260619 06:59:43.452446 23233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa877fb391f403f8e31a55a731a6f62" candidate_uuid: "8c38388f77ce4c54a93177fdc101a02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30be41cfd97464d9fb576893ce9a827" is_pre_election: true
I20260619 06:59:43.452836 23233 raft_consensus.cc:2454] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c38388f77ce4c54a93177fdc101a02d in term 0.
I20260619 06:59:43.453735 23300 leader_election.cc:302] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c38388f77ce4c54a93177fdc101a02d, c30be41cfd97464d9fb576893ce9a827; no voters:
I20260619 06:59:43.454177 23441 raft_consensus.cc:2781] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:59:43.454272 23441 raft_consensus.cc:482] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:59:43.454315 23441 raft_consensus.cc:3037] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:59:43.455725 23441 raft_consensus.cc:504] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } }
I20260619 06:59:43.456771 23233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa877fb391f403f8e31a55a731a6f62" candidate_uuid: "8c38388f77ce4c54a93177fdc101a02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30be41cfd97464d9fb576893ce9a827"
I20260619 06:59:43.456904 23233 raft_consensus.cc:3037] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:59:43.457227 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.446513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:43777 (local address 127.11.106.65:45545)
0619 06:59:43.447035 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:43.447039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:43.447053 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:59:43.447101 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:43.447104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:43.447148 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:43.447218 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:59:43.453260 (+ 6042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.454062 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.455704 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.456317 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.456428 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.456487 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:43.456998 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 06:59:43.457091 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435}
I20260619 06:59:43.457659 23443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.445438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45545 (local address 127.11.106.67:43777)
0619 06:59:43.446298 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 06:59:43.446316 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:59:43.446422 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:59:43.452834 (+ 6412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:59:43.452837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:59:43.452865 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:59:43.453024 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.453080 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.454284 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.454287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.455523 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:59:43.455545 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.456732 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.456735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:59:43.456838 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.456863 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:59:43.457374 (+ 511us) client_negotiation.cc:240] Negotiation successful
0619 06:59:43.457464 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":151,"threads_started":1}
I20260619 06:59:43.458277 23233 raft_consensus.cc:2454] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c38388f77ce4c54a93177fdc101a02d in term 1.
I20260619 06:59:43.459617 23089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa877fb391f403f8e31a55a731a6f62" candidate_uuid: "8c38388f77ce4c54a93177fdc101a02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd46dd4054746b38ddf9ea855eafbdb" is_pre_election: true
I20260619 06:59:43.459960 23089 raft_consensus.cc:2454] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c38388f77ce4c54a93177fdc101a02d in term 0.
I20260619 06:59:43.460305 23088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fa877fb391f403f8e31a55a731a6f62" candidate_uuid: "8c38388f77ce4c54a93177fdc101a02d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd46dd4054746b38ddf9ea855eafbdb"
I20260619 06:59:43.460404 23088 raft_consensus.cc:3037] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:59:43.461781 23088 raft_consensus.cc:2454] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c38388f77ce4c54a93177fdc101a02d in term 1.
I20260619 06:59:43.462828 23302 leader_election.cc:302] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c38388f77ce4c54a93177fdc101a02d, afd46dd4054746b38ddf9ea855eafbdb; no voters:
I20260619 06:59:43.463310 23441 leader_election.cc:288] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [CANDIDATE]: Term 1 election: Requested vote from peers c30be41cfd97464d9fb576893ce9a827 (127.11.106.66:44349), afd46dd4054746b38ddf9ea855eafbdb (127.11.106.65:45545)
I20260619 06:59:43.463382 23444 raft_consensus.cc:2781] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:59:43.463583 23444 raft_consensus.cc:686] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 1 LEADER]: Becoming Leader. State: Replica: 8c38388f77ce4c54a93177fdc101a02d, State: Running, Role: LEADER
I20260619 06:59:43.464021 23444 consensus_queue.cc:237] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } }
I20260619 06:59:43.469314 22923 catalog_manager.cc:5495] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c38388f77ce4c54a93177fdc101a02d (127.11.106.67). New cstate: current_term: 1 leader_uuid: "8c38388f77ce4c54a93177fdc101a02d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c38388f77ce4c54a93177fdc101a02d" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 } health_report { overall_health: UNKNOWN } } }
I20260619 06:59:43.518132 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.512380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.11.106.67:43499)
0619 06:59:43.512580 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:59:43.512585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:59:43.512695 (+ 110us) server_negotiation.cc:407] Connection header received
0619 06:59:43.512888 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:59:43.512892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:59:43.512972 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:59:43.513098 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:59:43.513514 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.514309 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.515579 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:59:43.515906 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:59:43.516009 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:59:43.516471 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:59:43.517310 (+ 839us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:59:43.517435 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:59:43.517916 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 06:59:43.517984 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:59:43.526398 23088 raft_consensus.cc:1261] T 5fa877fb391f403f8e31a55a731a6f62 P afd46dd4054746b38ddf9ea855eafbdb [term 1 FOLLOWER]: Refusing update from remote peer 8c38388f77ce4c54a93177fdc101a02d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:59:43.527381 23444 consensus_queue.cc:1035] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [LEADER]: Connected to new peer: Peer: permanent_uuid: "afd46dd4054746b38ddf9ea855eafbdb" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:59:43.529016 23233 raft_consensus.cc:1261] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [term 1 FOLLOWER]: Refusing update from remote peer 8c38388f77ce4c54a93177fdc101a02d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:59:43.529686 23441 consensus_queue.cc:1035] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30be41cfd97464d9fb576893ce9a827" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:59:43.538090 23447 mvcc.cc:205] Tried to move back new op lower bound from 7298467362913759232 to 7298467362676228096. Current Snapshot: MvccSnapshot[applied={T|T < 7298467362913759232}]
I20260619 06:59:43.539359 23450 mvcc.cc:205] Tried to move back new op lower bound from 7298467362913759232 to 7298467362676228096. Current Snapshot: MvccSnapshot[applied={T|T < 7298467362913759232}]
I20260619 06:59:43.547413 23448 mvcc.cc:205] Tried to move back new op lower bound from 7298467362913759232 to 7298467362676228096. Current Snapshot: MvccSnapshot[applied={T|T < 7298467362913759232}]
I20260619 06:59:43.559531 22929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:59:43.559746 22929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:59:43.562842 22929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57a0ade9a7364568b163684d39e4b4b3: Sending DeleteTablet for 3 replicas of tablet 5fa877fb391f403f8e31a55a731a6f62
I20260619 06:59:43.563787 23213 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa877fb391f403f8e31a55a731a6f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:59:43 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:45700
I20260619 06:59:43.564044 23065 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa877fb391f403f8e31a55a731a6f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:59:43 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:36868
I20260619 06:59:43.565081 23346 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fa877fb391f403f8e31a55a731a6f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:59:43 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50596
I20260619 06:59:43.565685 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 23010
I20260619 06:59:43.573050 23457 tablet_replica.cc:330] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: stopping tablet replica
I20260619 06:59:43.573402 23459 tablet_replica.cc:330] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: stopping tablet replica
I20260619 06:59:43.573959 23457 raft_consensus.cc:2229] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:59:43.574440 23459 raft_consensus.cc:2229] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:59:43.574910 23457 raft_consensus.cc:2258] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:59:43.575484 23459 raft_consensus.cc:2258] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:59:43.577759 23457 ts_tablet_manager.cc:1905] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:59:43.579347 23459 ts_tablet_manager.cc:1905] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:59:43.583818 23459 ts_tablet_manager.cc:1918] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:59:43.583818 23457 ts_tablet_manager.cc:1918] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:59:43.583956 23457 log.cc:1196] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/5fa877fb391f403f8e31a55a731a6f62
I20260619 06:59:43.583956 23459 log.cc:1196] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/5fa877fb391f403f8e31a55a731a6f62
I20260619 06:59:43.584360 23459 ts_tablet_manager.cc:1939] T 5fa877fb391f403f8e31a55a731a6f62 P 8c38388f77ce4c54a93177fdc101a02d: Deleting consensus metadata
I20260619 06:59:43.585482 23457 ts_tablet_manager.cc:1939] T 5fa877fb391f403f8e31a55a731a6f62 P c30be41cfd97464d9fb576893ce9a827: Deleting consensus metadata
W20260619 06:59:43.586076 22915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:45545 (error 108)
I20260619 06:59:43.586656 22915 catalog_manager.cc:4841] TS 8c38388f77ce4c54a93177fdc101a02d (127.11.106.67:43499): tablet 5fa877fb391f403f8e31a55a731a6f62 (table test-table [id=4441e660107b435db7e8a4f3f59f9089]) successfully deleted
I20260619 06:59:43.586973 22913 catalog_manager.cc:4841] TS c30be41cfd97464d9fb576893ce9a827 (127.11.106.66:44349): tablet 5fa877fb391f403f8e31a55a731a6f62 (table test-table [id=4441e660107b435db7e8a4f3f59f9089]) successfully deleted
I20260619 06:59:43.588073 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:59:43.587730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45545 (local address 127.0.0.1:36880)
0619 06:59:43.587916 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 06:59:43.587982 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:45545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260619 06:59:43.588354 22915 catalog_manager.cc:4584] TS afd46dd4054746b38ddf9ea855eafbdb (127.11.106.65:45545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fa877fb391f403f8e31a55a731a6f62: Network error: Client connection negotiation failed: client connection to 127.11.106.65:45545: connect: Connection refused (error 111)
I20260619 06:59:43.588632 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 23149
I20260619 06:59:43.596791 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 23284
I20260619 06:59:43.605046 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 22897
I20260619 06:59:43.612470 11689 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:59:43.947783 11689 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:59:43.947907 11689 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:59:44.292836 11689 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:59:44.293 UTC [22483] LOG: received smart shutdown request
2026-06-19 06:59:44.296 UTC [22483] LOG: background worker "logical replication launcher" (PID 22492) exited with exit code 1
2026-06-19 06:59:44.297 UTC [22487] LOG: shutting down
2026-06-19 06:59:44.472 UTC [22483] LOG: database system is shut down
2026-06-19T06:59:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47957 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:59:44 dist-test-slave-10d0 krb5kdc[23472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:59:44 dist-test-slave-10d0 krb5kdc[23472](info): set up 2 sockets
Jun 19 06:59:44 dist-test-slave-10d0 krb5kdc[23472](info): commencing operation
krb5kdc: starting...
W20260619 06:59:45.661273 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 06:59:45 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:59:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.106.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.106.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/rangeradmin_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/rangeradmin_127.11.106.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.106.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.106.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/rangerlookup_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/rangerlookup_127.11.106.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.106.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.106.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/HTTP_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/HTTP_127.11.106.84@KRBTEST.COM.keytab.
I20260619 06:59:45.827435 11689 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1_IlA7/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/postgres -l logfile start
2026-06-19 06:59:48.314 UTC [23502] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:59:48.314 UTC [23502] LOG: listening on IPv4 address "127.11.106.84", port 48945
2026-06-19 06:59:48.317 UTC [23502] LOG: listening on Unix socket "/tmp/.s.PGSQL.48945"
2026-06-19 06:59:48.344 UTC [23505] LOG: database system was shut down at 2026-06-19 06:59:46 UTC
2026-06-19 06:59:48.347 UTC [23502] LOG: database system is ready to accept connections
I20260619 06:59:49.365407 11689 mini_postgres.cc:96] Postgres bound to 48945
2026-06-19 06:59:49.371 UTC [23513] FATAL: database "slave" does not exist
127.11.106.84:48945 - accepting connections
I20260619 06:59:49.372288 11689 mini_ranger.cc:161] Starting Ranger...
I20260619 06:59:49.386924 11689 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:59:49.845396 11689 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:59:49.845471 11689 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-admin
2026-06-19 06:59:50,142 [I] DB FLAVOR :POSTGRES
2026-06-19 06:59:50,143 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:59:50,143 [I] Checking connection..
2026-06-19 06:59:50,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:59:50,404 [I] Checking connection passed.
2026-06-19 06:59:50,404 [I] --------- Verifying version history table ---------
2026-06-19 06:59:50,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:59:50,641 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:59:50,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:59:50,908 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:59:50,908 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:59:50,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:59:51.154 UTC [23577] WARNING: there is no transaction in progress
2026-06-19 06:59:51,161 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:59:51,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:59:51,409 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:59:51,409 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:59:51,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:59:51,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:59:51,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-10d0.c.gcp-upstream.internal','N') ;"
2026-06-19 06:59:52,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:59:52,378 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:59:52,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:59:52,610 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:59:52,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:59:52,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:59:53,102 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:59:53,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:59:54.100 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.116 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.133 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.149 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.165 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.183 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.246 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.253 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.267 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.278 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.289 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.303 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.313 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.320 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.328 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:54.335 UTC [23685] WARNING: there is no transaction in progress
2026-06-19 06:59:55,001 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:59:55,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:59:55,282 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:59:55,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:59:55,536 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:59:55,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:59:55,769 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:59:55,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:59:56,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:59:56,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-10d0.c.gcp-upstream.internal';"
2026-06-19 06:59:56,518 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:59:56,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:59:56,757 [I] DB_PATCHES have already been applied
I20260619 06:59:56.763099 11689 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-admin:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1_IlA7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:59:56.763162 11689 mini_ranger.cc:193] Using host: 127.11.106.84
I20260619 06:59:56.766710 11689 mini_ranger.cc:227] Ranger admin URL: http://127.11.106.84:45983
Jun 19, 2026 6:59:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:59:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:59:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:59:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:59:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:59:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:59:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:59:57.863049 11689 mini_ranger.cc:160] Time spent starting Ranger: real 8.491s user 0.002s sys 0.007s
Jun 19, 2026 7:00:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 7:00:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 7:00:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 7:00:18 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 7:00:19 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 07:00:19.783704 11689 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.106.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.106.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/rangerkms_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.106.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/rangerkms_127.11.106.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.106.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.106.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.106.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/HTTP_127.11.106.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.106.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/HTTP_127.11.106.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 07:00:19 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852419, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 07:00:19.882934 11689 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 07:00:19.892501 11689 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 07:00:20.756711 11689 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 07:00:20.756796 11689 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-kms
I20260619 07:00:20.756810 11689 mini_ranger_kms.cc:227] Using postgres at 127.11.106.84:48945
2026-06-19 07:00:20,926 [I] DB FLAVOR :POSTGRES
2026-06-19 07:00:20,926 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 07:00:20,926 [I] Checking connection
2026-06-19 07:00:20,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 07:00:21,173 [I] connection success
2026-06-19 07:00:21,173 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 07:00:21,173 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 07:00:21,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 07:00:21,438 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 07:00:21,438 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 07:00:21,438 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 07:00:21,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.106.84:48945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 07:00:21,689 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 07:00:22.191051 11689 mini_ranger_kms.cc:310] Created kms service
I20260619 07:00:22.412890 11689 mini_ranger_kms.cc:326] Created kudu user
I20260619 07:00:22.455307 11689 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 07:00:22 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852419, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:22.862545 11689 mini_ranger_kms.cc:384] Added ranger policy
I20260619 07:00:22.862704 11689 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-kms:/tmp/dist-test-task1_IlA7/build/debug/bin/postgresql.jar:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1_IlA7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 07:00:22.862727 11689 mini_ranger_kms.cc:240] Using host: 127.11.106.84
I20260619 07:00:22.866657 11689 mini_ranger_kms.cc:276] Ranger KMS PID: 23865
I20260619 07:00:22.866753 11689 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.106.84:57957
Jun 19, 2026 7:00:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 7:00:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 7:00:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 7:00:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 7:00:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 7:00:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 07:00:25.127597 11689 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.245s user 0.005s sys 0.148s
I20260619 07:00:25.127737 11689 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 07:00:25.127812 11689 mini_ranger_kms.cc:401] 127.11.106.84:57957/kms/v1/keys
Jun 19 07:00:25 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.106.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 07:00:25 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.106.84@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
Jun 19 07:00:25 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852425, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.106.84@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
Jun 19 07:00:28 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:00:28.131086 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:33571
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:33571
--ranger_config_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:57957/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:28.267671 23916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:28.267988 23916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:28.268028 23916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:28.272054 23916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:28.272115 23916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:28.272135 23916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:28.272151 23916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:28.272166 23916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:28.277637 23916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.106.84:57957/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:33571
--ranger_config_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:28.278716 23916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:28.279742 23916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:28.289741 23922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:28.289842 23916 server_base.cc:1034] running on GCE node
W20260619 07:00:28.290043 23924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:28.290027 23921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:28.290630 23916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:28.291802 23916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:28.292989 23916 hybrid_clock.cc:648] HybridClock initialized: now 1781852428292967 us; error 42 us; skew 500 ppm
Jun 19 07:00:28 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852428, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:28.302861 23916 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:00:28.303972 23916 webserver.cc:460] Webserver started at http://127.11.106.126:43265/ using document root <none> and password file <none>
I20260619 07:00:28.304647 23916 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:28.304735 23916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:28.304962 23916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 07:00:28 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852428, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:28.360617 23916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "d339b48de74b44c9a1361b6c8935af34"
format_stamp: "Formatted at 2026-06-19 07:00:28 on dist-test-slave-10d0"
server_key: "f6bdc22ab7aae02914bb0b355eed8a8e"
server_key_iv: "f245698a910ebeb29195b4b2bbf0f810"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:28.361325 23916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "d339b48de74b44c9a1361b6c8935af34"
format_stamp: "Formatted at 2026-06-19 07:00:28 on dist-test-slave-10d0"
server_key: "f6bdc22ab7aae02914bb0b355eed8a8e"
server_key_iv: "f245698a910ebeb29195b4b2bbf0f810"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:28.365438 23916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 07:00:28.380116 23931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:28.381652 23916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 07:00:28.381768 23916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "d339b48de74b44c9a1361b6c8935af34"
format_stamp: "Formatted at 2026-06-19 07:00:28 on dist-test-slave-10d0"
server_key: "f6bdc22ab7aae02914bb0b355eed8a8e"
server_key_iv: "f245698a910ebeb29195b4b2bbf0f810"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:28.381942 23916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:28.405977 23916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:28.409363 23916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:28.409577 23916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:28.417955 23916 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:33571
I20260619 07:00:28.417974 23983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:33571 every 8 connection(s)
I20260619 07:00:28.418991 23916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:00:28.420270 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 23916
I20260619 07:00:28.420403 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:00:28.420737 11689 external_mini_cluster.cc:1428] Setting key dc97e8009d80ca033e91211f74c7a0a4
I20260619 07:00:28.422461 23984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:28.429059 23984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Bootstrap starting.
Jun 19 07:00:28 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:28.431679 23984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:28.432536 23984 log.cc:824] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:28.434648 23984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: No bootstrap required, opened a new log
I20260619 07:00:28.438017 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:28.422263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.11.106.126:33571)
0619 07:00:28.422748 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:28.422766 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:28.422809 (+ 43us) server_negotiation.cc:407] Connection header received
0619 07:00:28.423642 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:28.423665 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:28.424074 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:28.424446 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:28.425358 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:28.426821 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:28.428010 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:28.428707 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:28.428860 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:28.432004 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:28.432011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:28.432030 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:28.432062 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:28.434735 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:28.435316 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:28.435322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:28.435328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:28.435461 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:28.435779 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:28.435782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:28.435783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:28.436278 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:28.436475 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:28.436800 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 07:00:28.437090 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":151,"threads_started":1}
I20260619 07:00:28.438211 23984 raft_consensus.cc:348] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } }
I20260619 07:00:28.438529 23984 raft_consensus.cc:374] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:28.438593 23984 raft_consensus.cc:729] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d339b48de74b44c9a1361b6c8935af34, State: Initialized, Role: FOLLOWER
I20260619 07:00:28.439059 23984 consensus_queue.cc:260] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } }
I20260619 07:00:28.439329 23984 raft_consensus.cc:388] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:00:28.439404 23984 raft_consensus.cc:482] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:00:28.439519 23984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:28.440670 23984 raft_consensus.cc:504] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } }
I20260619 07:00:28.440956 23984 leader_election.cc:302] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d339b48de74b44c9a1361b6c8935af34; no voters:
I20260619 07:00:28.441236 23984 leader_election.cc:288] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:00:28.441300 23989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:00:28.441502 23989 raft_consensus.cc:686] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 1 LEADER]: Becoming Leader. State: Replica: d339b48de74b44c9a1361b6c8935af34, State: Running, Role: LEADER
I20260619 07:00:28.442452 23989 consensus_queue.cc:237] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } }
I20260619 07:00:28.442517 23984 sys_catalog.cc:564] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:28.443663 23989 sys_catalog.cc:455] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d339b48de74b44c9a1361b6c8935af34. Latest consensus state: current_term: 1 leader_uuid: "d339b48de74b44c9a1361b6c8935af34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } } }
I20260619 07:00:28.443846 23989 sys_catalog.cc:458] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:28.444096 23990 sys_catalog.cc:455] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d339b48de74b44c9a1361b6c8935af34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } } }
I20260619 07:00:28.444193 23990 sys_catalog.cc:458] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:28.444293 23999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:00:28.444715 23984 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 07:00:28.451440 23999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:00:28.460669 24008 mvcc.cc:205] Tried to move back new op lower bound from 7298467546959982592 to 7298467546901143552. Current Snapshot: MvccSnapshot[applied={T|T < 7298467546959982592}]
I20260619 07:00:28.462417 23999 catalog_manager.cc:1347] Generated new cluster ID: c3ae4701c3664d89a7ea51bb57c04398
I20260619 07:00:28.462476 23999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:00:28.484364 23999 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:00:28.485795 23999 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:00:28.496908 23999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Generated new TSK 0
I20260619 07:00:28.497917 23999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 07:00:29.173475 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:57957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33571
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:29.318352 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:29.318636 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:29.318681 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:29.323957 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:29.324040 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:29.324131 24029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:00:29.332893 24029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.106.84:57957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:29.334333 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:29.335811 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:29.346783 24037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:29.346879 24039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:29.346783 24036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:29.347208 24029 server_base.cc:1034] running on GCE node
I20260619 07:00:29.347693 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:29.348351 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:29.349560 24029 hybrid_clock.cc:648] HybridClock initialized: now 1781852429349532 us; error 43 us; skew 500 ppm
Jun 19 07:00:29 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:29.352401 24029 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:00:29.353319 24029 webserver.cc:460] Webserver started at http://127.11.106.65:41497/ using document root <none> and password file <none>
I20260619 07:00:29.353864 24029 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:29.353920 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:29.354154 24029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 07:00:29 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:29.372496 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "46eea7a24dad450fb3763f33b709fba0"
format_stamp: "Formatted at 2026-06-19 07:00:29 on dist-test-slave-10d0"
server_key: "5de17891a467e6c88340c3fe272ed1f5"
server_key_iv: "dcbbcef02826d45fef2232fe95687833"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:29.373621 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "46eea7a24dad450fb3763f33b709fba0"
format_stamp: "Formatted at 2026-06-19 07:00:29 on dist-test-slave-10d0"
server_key: "5de17891a467e6c88340c3fe272ed1f5"
server_key_iv: "dcbbcef02826d45fef2232fe95687833"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:29.382056 24029 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260619 07:00:29.399011 24046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:29.400856 24029 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 07:00:29.400987 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "46eea7a24dad450fb3763f33b709fba0"
format_stamp: "Formatted at 2026-06-19 07:00:29 on dist-test-slave-10d0"
server_key: "5de17891a467e6c88340c3fe272ed1f5"
server_key_iv: "dcbbcef02826d45fef2232fe95687833"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:29.401120 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:29.450865 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:29.454821 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:29.455102 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:29.455935 24029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:29.457118 24029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:29.457232 24029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:29.457363 24029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:29.457448 24029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:29.490221 24029 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:38747
I20260619 07:00:29.490401 24161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:38747 every 8 connection(s)
I20260619 07:00:29.491348 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:00:29.495401 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 24029
I20260619 07:00:29.495558 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 07:00:29.495927 11689 external_mini_cluster.cc:1428] Setting key 77cb52bb8e4dcce2a96ae9d40d04fbdf
Jun 19 07:00:29 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 07:00:29.510201 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:29.493599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:54601 (local address 127.11.106.126:33571)
0619 07:00:29.493894 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:29.493921 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:29.494830 (+ 909us) server_negotiation.cc:407] Connection header received
0619 07:00:29.496008 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:29.496011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:29.496075 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:29.496208 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:29.497965 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:29.498943 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:29.500958 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:29.501430 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:29.501534 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:29.505155 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:29.505162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:29.505165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:29.505198 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:29.507447 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:29.508174 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:29.508178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:29.508180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:29.508282 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:29.508798 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:29.508801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:29.508803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:29.509011 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:29.509150 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:29.509777 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 07:00:29.509914 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 07:00:29.511801 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:29.494149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:33571 (local address 127.11.106.65:54601)
0619 07:00:29.494666 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:29.494724 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:29.495770 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:29.496427 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:29.496436 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:29.497071 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:29.497789 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:29.497810 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:29.499130 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:29.499137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:29.500797 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:29.500811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:29.501770 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:29.501777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:29.502043 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:29.502817 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:29.502847 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:29.504916 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:29.507615 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:29.507623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:29.507641 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:29.508028 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:29.508411 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:29.508414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:29.508416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:29.508672 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:29.509212 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:29.509223 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:29.509502 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 07:00:29.509718 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 07:00:29.510122 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":152,"threads_started":1}
I20260619 07:00:29.513720 24162 heartbeater.cc:347] Connected to a master server at 127.11.106.126:33571
I20260619 07:00:29.514001 24162 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:29.514655 24162 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
I20260619 07:00:29.516700 23948 ts_manager.cc:194] Registered new tserver with Master: 46eea7a24dad450fb3763f33b709fba0 (127.11.106.65:38747)
I20260619 07:00:29.518803 23948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:54601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 07:00:29.529618 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:57957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33571
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 07:00:29 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 07:00:29.657358 24168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:29.657662 24168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:29.657780 24168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:29.661934 24168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:29.662019 24168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:29.662124 24168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:00:29.668030 24168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.106.84:57957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:29.669240 24168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:29.670284 24168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:29.679992 24177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:29.679992 24174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:29.680086 24175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:29.681265 24168 server_base.cc:1034] running on GCE node
I20260619 07:00:29.681717 24168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:29.682441 24168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:29.683629 24168 hybrid_clock.cc:648] HybridClock initialized: now 1781852429683598 us; error 48 us; skew 500 ppm
Jun 19 07:00:29 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:29.686820 24168 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:00:29.687716 24168 webserver.cc:460] Webserver started at http://127.11.106.66:38397/ using document root <none> and password file <none>
I20260619 07:00:29.688298 24168 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:29.688393 24168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:29.688695 24168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 07:00:29 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:29.701838 24168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "78ed1f84b259439981b1d7cdb1352e90"
format_stamp: "Formatted at 2026-06-19 07:00:29 on dist-test-slave-10d0"
server_key: "d2fa0a24bcb915ed4380fc0bd5b1458e"
server_key_iv: "65b18510bca5b2157620534916b3f384"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:29.702636 24168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "78ed1f84b259439981b1d7cdb1352e90"
format_stamp: "Formatted at 2026-06-19 07:00:29 on dist-test-slave-10d0"
server_key: "d2fa0a24bcb915ed4380fc0bd5b1458e"
server_key_iv: "65b18510bca5b2157620534916b3f384"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:29.712296 24168 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260619 07:00:29.736326 24184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:29.738492 24168 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260619 07:00:29.738731 24168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "78ed1f84b259439981b1d7cdb1352e90"
format_stamp: "Formatted at 2026-06-19 07:00:29 on dist-test-slave-10d0"
server_key: "d2fa0a24bcb915ed4380fc0bd5b1458e"
server_key_iv: "65b18510bca5b2157620534916b3f384"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:29.738983 24168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:29.760774 24168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:29.764747 24168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:29.765027 24168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:29.765868 24168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:29.767050 24168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:29.767159 24168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:29.767261 24168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:29.767333 24168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:29.779068 24168 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:38301
I20260619 07:00:29.779596 24297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:38301 every 8 connection(s)
I20260619 07:00:29.780404 24168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:00:29.789892 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 24168
I20260619 07:00:29.790073 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 07:00:29.790455 11689 external_mini_cluster.cc:1428] Setting key f8d0200e96933fc769aad621ff9b6fa4
Jun 19 07:00:29 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 07:00:29.809971 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:29.784422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:52611 (local address 127.11.106.126:33571)
0619 07:00:29.784598 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:29.784603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:29.784623 (+ 20us) server_negotiation.cc:407] Connection header received
0619 07:00:29.785401 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:29.785406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:29.785467 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:29.785531 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:29.787305 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:29.788176 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:29.791763 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:29.792618 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:29.792714 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:29.804348 (+ 11634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:29.804354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:29.804357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:29.804392 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:29.807005 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:29.807789 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:29.807792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:29.807794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:29.807893 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:29.808397 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:29.808402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:29.808404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:29.808710 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:29.808810 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:29.809641 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 07:00:29.809787 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 07:00:29.812165 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:29.783243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:33571 (local address 127.11.106.66:52611)
0619 07:00:29.783843 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:29.783903 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:29.785050 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:29.785989 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:29.785999 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:29.786429 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:29.787109 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:29.787130 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:29.788361 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:29.788364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:29.791592 (+ 3228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:29.791605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:29.792943 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:29.792948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:29.797133 (+ 4185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:29.799040 (+ 1907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:29.799066 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:29.804054 (+ 4988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:29.807173 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:29.807180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:29.807205 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:29.807641 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:29.808068 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:29.808071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:29.808073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:29.808267 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:29.808886 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:29.808896 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:29.809260 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 07:00:29.810404 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0619 07:00:29.810852 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":144,"threads_started":1}
I20260619 07:00:29.813889 24298 heartbeater.cc:347] Connected to a master server at 127.11.106.126:33571
I20260619 07:00:29.814145 24298 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:29.814757 24298 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 07:00:29.816058 23948 ts_manager.cc:194] Registered new tserver with Master: 78ed1f84b259439981b1d7cdb1352e90 (127.11.106.66:38301)
I20260619 07:00:29.816784 23948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:52611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 07:00:29.833784 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:57957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33571
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:30.001096 24303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:30.001463 24303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:30.001528 24303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:30.007593 24303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:30.007726 24303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:30.007818 24303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:00:30.016078 24303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.106.84:57957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:30.017654 24303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:30.018929 24303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:30.030508 24308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:30.031020 24311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:30.031790 24309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:30.036569 24303 server_base.cc:1034] running on GCE node
I20260619 07:00:30.037278 24303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:30.038386 24303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:30.042013 24303 hybrid_clock.cc:648] HybridClock initialized: now 1781852430041762 us; error 264 us; skew 500 ppm
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:30.046365 24303 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:00:30.048341 24303 webserver.cc:460] Webserver started at http://127.11.106.67:42975/ using document root <none> and password file <none>
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852429, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:30.048991 24303 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:30.049098 24303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:30.049326 24303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:30.071661 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "2881cd02a8264a719028641492be16ba"
format_stamp: "Formatted at 2026-06-19 07:00:30 on dist-test-slave-10d0"
server_key: "de9990921127ea5b399c50dcfacd8dae"
server_key_iv: "ccee7d14f3f4fc48fc4e2695269b9ee6"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:30.072427 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "2881cd02a8264a719028641492be16ba"
format_stamp: "Formatted at 2026-06-19 07:00:30 on dist-test-slave-10d0"
server_key: "de9990921127ea5b399c50dcfacd8dae"
server_key_iv: "ccee7d14f3f4fc48fc4e2695269b9ee6"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:30.079777 24303 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260619 07:00:30.121052 24318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:30.123083 24303 fs_manager.cc:725] Time spent opening block manager: real 0.024s user 0.001s sys 0.001s
I20260619 07:00:30.123252 24303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "2881cd02a8264a719028641492be16ba"
format_stamp: "Formatted at 2026-06-19 07:00:30 on dist-test-slave-10d0"
server_key: "de9990921127ea5b399c50dcfacd8dae"
server_key_iv: "ccee7d14f3f4fc48fc4e2695269b9ee6"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:30.123401 24303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:30.166781 24303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:30.170076 24303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:30.170282 24303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:30.170974 24303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:30.171944 24303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:30.172047 24303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:30.172142 24303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:30.172176 24303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:30.183367 24303 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:40997
I20260619 07:00:30.183535 24431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:40997 every 8 connection(s)
I20260619 07:00:30.184396 24303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:00:30.188750 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 24303
I20260619 07:00:30.188885 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 07:00:30.189227 11689 external_mini_cluster.cc:1428] Setting key f4b3bab83b0dc07113b67af6d0e7a784
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:30.200946 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:30.186981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:33571 (local address 127.11.106.67:58423)
0619 07:00:30.187491 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:30.187566 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:30.188573 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:30.189423 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:30.189432 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:30.189796 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:30.190508 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:30.190541 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:30.191417 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:30.191420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:30.192351 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:30.192360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:30.193031 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:30.193035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:30.193217 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:30.193810 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:30.193827 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:30.195181 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:30.197504 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:30.197511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:30.197528 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:30.197842 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:30.198191 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:30.198196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:30.198201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:30.198375 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:30.198842 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:30.198851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:30.199185 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 07:00:30.199410 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 07:00:30.199778 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":150,"threads_started":1}
I20260619 07:00:30.201493 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:30.186468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:58423 (local address 127.11.106.126:33571)
0619 07:00:30.186615 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:30.186620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:30.187674 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 07:00:30.188849 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:30.188853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:30.188917 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:30.188977 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:30.190677 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:30.191257 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:30.192483 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:30.192860 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:30.192988 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:30.195635 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:30.195639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:30.195641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:30.195681 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:30.197350 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:30.197958 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:30.197971 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:30.197973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:30.198069 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:30.198509 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:30.198515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:30.198517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:30.198717 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:30.198830 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:30.201210 (+ 2380us) server_negotiation.cc:299] Negotiation successful
0619 07:00:30.201344 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 07:00:30.202636 24432 heartbeater.cc:347] Connected to a master server at 127.11.106.126:33571
I20260619 07:00:30.202905 24432 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:30.203495 24432 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
I20260619 07:00:30.204767 23948 ts_manager.cc:194] Registered new tserver with Master: 2881cd02a8264a719028641492be16ba (127.11.106.67:40997)
I20260619 07:00:30.205515 23948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:58423
I20260619 07:00:30.215324 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:00:30.216051 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 24029
I20260619 07:00:30.224109 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 24168
I20260619 07:00:30.232759 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 24303
I20260619 07:00:30.238905 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 23916
I20260619 07:00:30.246052 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:33571
--webserver_interface=127.11.106.126
--webserver_port=43265
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:33571
--ranger_config_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:57957/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.106.84:57957/kms with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:30.357895 24435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:30.358211 24435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:30.358322 24435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:30.362468 24435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:30.362561 24435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:30.362605 24435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:30.362671 24435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:30.362715 24435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:30.368124 24435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.106.84:57957/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:33571
--ranger_config_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=43265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:30.369313 24435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:30.370381 24435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:30.378893 24440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:30.378894 24441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:30.378937 24443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:30.379264 24435 server_base.cc:1034] running on GCE node
I20260619 07:00:30.379739 24435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:30.380882 24435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:30.382038 24435 hybrid_clock.cc:648] HybridClock initialized: now 1781852430382016 us; error 47 us; skew 500 ppm
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:30.384840 24435 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:00:30.385741 24435 webserver.cc:460] Webserver started at http://127.11.106.126:43265/ using document root <none> and password file <none>
I20260619 07:00:30.386317 24435 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:30.386415 24435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:30.463410 24435 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:00:30.465746 24451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:30.467111 24435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:00:30.467248 24435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "d339b48de74b44c9a1361b6c8935af34"
format_stamp: "Formatted at 2026-06-19 07:00:28 on dist-test-slave-10d0"
server_key: "f6bdc22ab7aae02914bb0b355eed8a8e"
server_key_iv: "f245698a910ebeb29195b4b2bbf0f810"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:30.467796 24435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:30.489192 24435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:30.491852 24435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:30.492074 24435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:30.499734 24435 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:33571
I20260619 07:00:30.499764 24503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:33571 every 8 connection(s)
I20260619 07:00:30.500840 24435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:00:30.503957 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 24435
I20260619 07:00:30.504427 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:57957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:38747
--local_ip_for_outbound_sockets=127.11.106.65
--tserver_master_addrs=127.11.106.126:33571
--webserver_port=41497
--webserver_interface=127.11.106.65
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.106.84:57957/kms with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:00:30.505400 24504 sys_catalog.cc:263] Verifying existing consensus state
I20260619 07:00:30.507032 24504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Bootstrap starting.
I20260619 07:00:30.523388 24504 log.cc:824] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:30.529846 24504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 07:00:30.530246 24504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Bootstrap complete.
I20260619 07:00:30.534343 24504 raft_consensus.cc:348] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } }
I20260619 07:00:30.534706 24504 raft_consensus.cc:729] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d339b48de74b44c9a1361b6c8935af34, State: Initialized, Role: FOLLOWER
I20260619 07:00:30.535195 24504 consensus_queue.cc:260] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } }
I20260619 07:00:30.535310 24504 raft_consensus.cc:388] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:00:30.535394 24504 raft_consensus.cc:482] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:00:30.535490 24504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 1 FOLLOWER]: Advancing to term 2
I20260619 07:00:30.537406 24504 raft_consensus.cc:504] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } }
I20260619 07:00:30.537742 24504 leader_election.cc:302] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d339b48de74b44c9a1361b6c8935af34; no voters:
I20260619 07:00:30.538064 24504 leader_election.cc:288] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 07:00:30.538143 24508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 07:00:30.538403 24508 raft_consensus.cc:686] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [term 2 LEADER]: Becoming Leader. State: Replica: d339b48de74b44c9a1361b6c8935af34, State: Running, Role: LEADER
I20260619 07:00:30.538750 24508 consensus_queue.cc:237] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } }
I20260619 07:00:30.539280 24504 sys_catalog.cc:564] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:30.540092 24509 sys_catalog.cc:455] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d339b48de74b44c9a1361b6c8935af34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } } }
I20260619 07:00:30.540242 24509 sys_catalog.cc:458] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:30.540575 24510 sys_catalog.cc:455] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d339b48de74b44c9a1361b6c8935af34. Latest consensus state: current_term: 2 leader_uuid: "d339b48de74b44c9a1361b6c8935af34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d339b48de74b44c9a1361b6c8935af34" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33571 } } }
I20260619 07:00:30.540733 24510 sys_catalog.cc:458] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34 [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:30.541064 24517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:00:30.541519 24504 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 07:00:30.547101 24517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:00:30.547926 24517 catalog_manager.cc:1259] Loaded cluster ID: c3ae4701c3664d89a7ea51bb57c04398
I20260619 07:00:30.548020 24517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:00:30.550755 24517 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:00:30.551716 24517 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Loaded TSK: 0
I20260619 07:00:30.553088 24517 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 07:00:30.644927 24506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:30.645298 24506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:30.645396 24506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:30.650779 24506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:30.650863 24506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:30.650969 24506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:00:30.657155 24506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.106.84:57957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:38747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=41497
--tserver_master_addrs=127.11.106.126:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:30.658239 24506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:30.659271 24506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:30.669364 24542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:30.669431 24544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:30.669430 24541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:30.670212 24506 server_base.cc:1034] running on GCE node
I20260619 07:00:30.670738 24506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:30.671424 24506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:30.672678 24506 hybrid_clock.cc:648] HybridClock initialized: now 1781852430672649 us; error 44 us; skew 500 ppm
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:30.675758 24506 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:00:30.676868 24506 webserver.cc:460] Webserver started at http://127.11.106.65:41497/ using document root <none> and password file <none>
I20260619 07:00:30.677639 24506 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:30.677765 24506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:30.701960 24506 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 07:00:30.707046 24552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:30.708694 24506 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260619 07:00:30.708837 24506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "46eea7a24dad450fb3763f33b709fba0"
format_stamp: "Formatted at 2026-06-19 07:00:29 on dist-test-slave-10d0"
server_key: "5de17891a467e6c88340c3fe272ed1f5"
server_key_iv: "dcbbcef02826d45fef2232fe95687833"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:30.709573 24506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:30.741268 24506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:30.745179 24506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:30.745390 24506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:30.746177 24506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:30.747272 24506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:30.747339 24506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:30.747396 24506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:30.747467 24506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 07:00:30.758899 24506 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:38747
I20260619 07:00:30.758966 24665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:38747 every 8 connection(s)
I20260619 07:00:30.760345 24506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:00:30.764863 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 24506
I20260619 07:00:30.765375 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:57957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:38301
--local_ip_for_outbound_sockets=127.11.106.66
--tserver_master_addrs=127.11.106.126:33571
--webserver_port=38397
--webserver_interface=127.11.106.66
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.106.84:57957/kms with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:30.789731 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:30.765025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:50877 (local address 127.11.106.126:33571)
0619 07:00:30.767341 (+ 2316us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:30.767362 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:30.767406 (+ 44us) server_negotiation.cc:407] Connection header received
0619 07:00:30.767707 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:30.767723 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:30.768120 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:30.768464 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:30.770188 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:30.771833 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:30.773725 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:30.774454 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:30.774640 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:30.777799 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:30.777809 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:30.777829 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:30.777863 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:30.785357 (+ 7494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:30.786140 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:30.786149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:30.786158 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:30.786366 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:30.786839 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:30.786843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:30.786845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:30.787373 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:30.787748 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:30.788306 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 07:00:30.788681 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2152,"thread_start_us":112,"threads_started":1}
I20260619 07:00:30.792629 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:30.763549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:33571 (local address 127.11.106.65:50877)
0619 07:00:30.765750 (+ 2201us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:30.765805 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:30.766840 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:30.768984 (+ 2144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:30.768993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:30.769339 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:30.769983 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:30.770007 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:30.771995 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:30.771999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:30.773580 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:30.773591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:30.774863 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:30.774868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:30.775067 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:30.775806 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:30.775831 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:30.777580 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:30.785522 (+ 7942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:30.785537 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:30.785554 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:30.785958 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:30.786494 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:30.786499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:30.786501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:30.786706 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:30.787517 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:30.787536 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:30.787882 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 07:00:30.791326 (+ 3444us) client_negotiation.cc:240] Negotiation successful
0619 07:00:30.791712 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1896,"thread_start_us":160,"threads_started":1}
I20260619 07:00:30.796111 24666 heartbeater.cc:347] Connected to a master server at 127.11.106.126:33571
I20260619 07:00:30.796347 24666 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:30.797015 24666 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
W20260619 07:00:30.799317 24666 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:33571 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 07:00:30.799434 24666 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:30.799603 24666 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
I20260619 07:00:30.800489 24666 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:30.800673 24666 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
W20260619 07:00:30.801434 24666 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 07:00:30.921331 24670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:30.921684 24670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:30.921784 24670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:30.926669 24670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:30.926759 24670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:30.926851 24670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:00:30.933622 24670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.106.84:57957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:38301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=38397
--tserver_master_addrs=127.11.106.126:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:30.935534 24670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:30.936690 24670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:30.946702 24675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:30.946939 24678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:30.947297 24676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:30.947727 24670 server_base.cc:1034] running on GCE node
I20260619 07:00:30.948155 24670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:30.948897 24670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:30.950083 24670 hybrid_clock.cc:648] HybridClock initialized: now 1781852430950055 us; error 39 us; skew 500 ppm
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:30.952857 24670 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:00:30.953724 24670 webserver.cc:460] Webserver started at http://127.11.106.66:38397/ using document root <none> and password file <none>
I20260619 07:00:30.954277 24670 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:30.954345 24670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 07:00:30 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:30.979029 24670 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260619 07:00:30.982195 24686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:30.983560 24670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 07:00:30.983727 24670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "78ed1f84b259439981b1d7cdb1352e90"
format_stamp: "Formatted at 2026-06-19 07:00:29 on dist-test-slave-10d0"
server_key: "d2fa0a24bcb915ed4380fc0bd5b1458e"
server_key_iv: "65b18510bca5b2157620534916b3f384"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:30.984285 24670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:30.996551 24670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:31.000339 24670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:31.000610 24670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:31.001413 24670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:31.002565 24670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:31.002636 24670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:31.002698 24670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:31.002776 24670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 07:00:31.014886 24670 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:38301
I20260619 07:00:31.016427 24670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:00:31.017743 24799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:38301 every 8 connection(s)
I20260619 07:00:31.024135 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 24670
I20260619 07:00:31.024749 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.106.84:57957/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:40997
--local_ip_for_outbound_sockets=127.11.106.67
--tserver_master_addrs=127.11.106.126:33571
--webserver_port=42975
--webserver_interface=127.11.106.67
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.106.84:57957/kms with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 07:00:31 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852430, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:31.038539 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:31.020990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:33571 (local address 127.11.106.66:40221)
0619 07:00:31.021636 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:31.021698 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:31.022974 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:31.023662 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:31.023675 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:31.024087 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:31.024895 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:31.024923 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.026256 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.026261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:31.027296 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:31.027306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.028237 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.028242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:31.028484 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:31.029314 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:31.029343 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:31.031311 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:31.034106 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:31.034118 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:31.034143 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:31.034610 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:31.035003 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:31.035007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:31.035009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:31.035250 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:31.035903 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:31.035914 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:31.036324 (+ 410us) client_negotiation.cc:769] Sending connection context
0619 07:00:31.036623 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 07:00:31.037083 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":140,"threads_started":1}
I20260619 07:00:31.039554 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:31.020356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:40221 (local address 127.11.106.126:33571)
0619 07:00:31.020502 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:31.020508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:31.021825 (+ 1317us) server_negotiation.cc:407] Connection header received
0619 07:00:31.023255 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:31.023259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:31.023326 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:31.023426 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:31.025112 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.026056 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.027449 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.027905 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.028028 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:31.031591 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:31.031597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:31.031605 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:31.031646 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:31.033934 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:31.034759 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:31.034764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:31.034768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:31.034875 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:31.035391 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:31.035397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:31.035402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:31.035721 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:31.035835 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:31.039045 (+ 3210us) server_negotiation.cc:299] Negotiation successful
0619 07:00:31.039226 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 07:00:31.041747 24800 heartbeater.cc:347] Connected to a master server at 127.11.106.126:33571
I20260619 07:00:31.042055 24800 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:31.042627 24800 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
W20260619 07:00:31.044370 24800 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:33571 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 07:00:31.044489 24800 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:31.044690 24800 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
I20260619 07:00:31.045567 24800 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:31.045739 24800 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
W20260619 07:00:31.046509 24800 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 07:00:31.176215 24803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:31.176533 24803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:31.176637 24803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:31.181574 24803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:31.181653 24803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:31.181751 24803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:00:31.189517 24803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.106.84:57957/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:40997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=42975
--tserver_master_addrs=127.11.106.126:33571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:31.191010 24803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:31.192473 24803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:31.203547 24809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:31.203817 24812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:31.203547 24810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:31.204484 24803 server_base.cc:1034] running on GCE node
I20260619 07:00:31.204924 24803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:31.205708 24803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:31.213199 24803 hybrid_clock.cc:648] HybridClock initialized: now 1781852431213163 us; error 67 us; skew 500 ppm
Jun 19 07:00:31 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:31.217031 24803 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:00:31.218274 24803 webserver.cc:460] Webserver started at http://127.11.106.67:42975/ using document root <none> and password file <none>
I20260619 07:00:31.218910 24803 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:31.219024 24803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 07:00:31 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:31.238606 24803 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 07:00:31.242395 24820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:31.245128 24803 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260619 07:00:31.245342 24803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "2881cd02a8264a719028641492be16ba"
format_stamp: "Formatted at 2026-06-19 07:00:30 on dist-test-slave-10d0"
server_key: "de9990921127ea5b399c50dcfacd8dae"
server_key_iv: "ccee7d14f3f4fc48fc4e2695269b9ee6"
server_key_version: "kuduclusterkey@0"
I20260619 07:00:31.246136 24803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:31.271343 24803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:31.275512 24803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:31.275722 24803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:31.276464 24803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:31.277586 24803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:31.277652 24803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:31.277709 24803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:31.277742 24803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:31.289983 24803 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:40997
I20260619 07:00:31.290125 24933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:40997 every 8 connection(s)
I20260619 07:00:31.291318 24803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:00:31.296201 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 24803
Jun 19 07:00:31 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:31.308597 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:31.295102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:42447 (local address 127.11.106.126:33571)
0619 07:00:31.295219 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:31.295224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:31.295241 (+ 17us) server_negotiation.cc:407] Connection header received
0619 07:00:31.296013 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:31.296016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:31.296094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:31.296157 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:31.298491 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.299392 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.301186 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.301548 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.301624 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:31.304455 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:31.304458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:31.304461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:31.304484 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:31.306243 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:31.306873 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:31.306877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:31.306878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:31.306962 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:31.307367 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:31.307371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:31.307373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:31.307602 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:31.307657 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:31.308281 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 07:00:31.308420 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 07:00:31.310051 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:31.293977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:33571 (local address 127.11.106.67:42447)
0619 07:00:31.294595 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:31.294655 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:31.295737 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:31.297215 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:31.297225 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:31.297588 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:31.298304 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:31.298325 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.299552 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.299556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:31.301027 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:31.301039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.301832 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.301836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:31.302020 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:31.302695 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:31.302715 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:31.304279 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:31.306377 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:31.306386 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:31.306406 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:31.306753 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:31.307081 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:31.307084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:31.307086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:31.307259 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:31.307724 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:31.307729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:31.307953 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 07:00:31.308828 (+ 875us) client_negotiation.cc:240] Negotiation successful
0619 07:00:31.309133 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":152,"threads_started":1}
I20260619 07:00:31.311306 24934 heartbeater.cc:347] Connected to a master server at 127.11.106.126:33571
I20260619 07:00:31.311570 24934 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:31.312178 24934 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
I20260619 07:00:31.313871 24937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:31.296887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.11.106.126:33571)
0619 07:00:31.304216 (+ 7329us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:31.304220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:31.304233 (+ 13us) server_negotiation.cc:407] Connection header received
0619 07:00:31.304268 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:31.304271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:31.304314 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:31.304395 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:31.305510 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.306355 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.307559 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:31.307913 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:31.308047 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:31.309686 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:31.309689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:31.309692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:31.309715 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:31.312018 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:31.312586 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:31.312590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:31.312592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:31.312676 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:31.312985 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:31.312988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:31.312990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:31.313204 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:31.313391 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:31.313596 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 07:00:31.313712 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7244,"thread_start_us":87,"threads_started":1}
W20260619 07:00:31.313872 24934 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:33571 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 07:00:31.314011 24934 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:31.314213 24934 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
I20260619 07:00:31.315164 24934 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:31.315327 24934 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
W20260619 07:00:31.316113 24934 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 07:00:31 dist-test-slave-10d0 krb5kdc[23472](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781852431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:31.801684 24666 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:31.801977 24666 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
I20260619 07:00:31.803241 24468 ts_manager.cc:194] Registered new tserver with Master: 46eea7a24dad450fb3763f33b709fba0 (127.11.106.65:38747)
I20260619 07:00:31.804987 24468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:50877
Jun 19 07:00:32 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:32.046739 24800 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:32.047030 24800 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
I20260619 07:00:32.047780 24467 ts_manager.cc:194] Registered new tserver with Master: 78ed1f84b259439981b1d7cdb1352e90 (127.11.106.66:38301)
I20260619 07:00:32.048616 24467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:40221
Jun 19 07:00:32 dist-test-slave-10d0 krb5kdc[23472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852431, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for HTTP/127.11.106.84@KRBTEST.COM
I20260619 07:00:32.316351 24934 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:32.316715 24934 heartbeater.cc:511] Master 127.11.106.126:33571 requested a full tablet report, sending...
I20260619 07:00:32.317545 24467 ts_manager.cc:194] Registered new tserver with Master: 2881cd02a8264a719028641492be16ba (127.11.106.67:40997)
I20260619 07:00:32.318341 24467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:42447
I20260619 07:00:32.325429 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:00:32.340878 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.327604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.11.106.126:33571)
0619 07:00:32.328272 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:32.328278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:32.328586 (+ 308us) server_negotiation.cc:407] Connection header received
0619 07:00:32.328812 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:32.328815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:32.328870 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:32.328957 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:32.330037 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.330855 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.332129 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.332518 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.332651 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.336200 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:32.336205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:32.336208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:32.336238 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:32.338408 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:32.339062 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:32.339066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:32.339068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:32.339170 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:32.339596 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:32.339599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:32.339601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:32.339889 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:32.340036 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:32.340451 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 07:00:32.340691 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":116,"threads_started":1}
I20260619 07:00:32.351554 24467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:00:32.356449 24467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:00:32.382695 24963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.374332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:38301 (local address 127.0.0.1:58812)
0619 07:00:32.376444 (+ 2112us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:32.376457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:32.376703 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:32.377421 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:32.377427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:32.377451 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:32.377590 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.377596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.379323 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.379326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.380782 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.380795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.382115 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.382118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.382225 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.382266 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 07:00:32.382378 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 07:00:32.382505 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2052,"thread_start_us":124,"threads_started":1}
I20260619 07:00:32.383424 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.374500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.11.106.66:38301)
0619 07:00:32.376294 (+ 1794us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:32.376301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:32.376630 (+ 329us) server_negotiation.cc:407] Connection header received
0619 07:00:32.376860 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:32.376866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:32.377085 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:32.377235 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:32.377722 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.379184 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.380929 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.381779 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.381895 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.382048 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:32.383168 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0619 07:00:32.383276 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1686,"thread_start_us":64,"threads_started":1}
I20260619 07:00:32.383821 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.371574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:40997 (local address 127.0.0.1:42800)
0619 07:00:32.375608 (+ 4034us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:32.375666 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:32.375824 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:32.378117 (+ 2293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:32.378120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:32.378140 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:32.378265 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.378273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.379753 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.379756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.381921 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.381934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.383306 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.383311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.383409 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.383436 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 07:00:32.383596 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 07:00:32.383670 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3817,"thread_start_us":120,"threads_started":1}
I20260619 07:00:32.384727 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.372505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42800 (local address 127.11.106.67:40997)
0619 07:00:32.376152 (+ 3647us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:32.376161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:32.377459 (+ 1298us) server_negotiation.cc:407] Connection header received
0619 07:00:32.377572 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:32.377577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:32.377793 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:32.377952 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:32.378420 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.379596 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.382084 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.382925 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.384251 (+ 1326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.384384 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:32.384429 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 07:00:32.384514 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3522,"thread_start_us":91,"threads_started":1}
I20260619 07:00:32.387028 24733 tablet_service.cc:1467] Processing CreateTablet for tablet 015a22a6e3e04344b72304165bc490ed (DEFAULT_TABLE table=test-table [id=edf1e7c90f7644229c10a90946f58214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:32.388293 24733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015a22a6e3e04344b72304165bc490ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:32.388564 24867 tablet_service.cc:1467] Processing CreateTablet for tablet 015a22a6e3e04344b72304165bc490ed (DEFAULT_TABLE table=test-table [id=edf1e7c90f7644229c10a90946f58214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:32.389657 24867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015a22a6e3e04344b72304165bc490ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:32.398980 24970 tablet_bootstrap.cc:492] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: Bootstrap starting.
I20260619 07:00:32.401558 24970 tablet_bootstrap.cc:654] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:32.402560 24970 log.cc:824] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:32.403395 24969 tablet_bootstrap.cc:492] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: Bootstrap starting.
I20260619 07:00:32.405637 24970 tablet_bootstrap.cc:492] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: No bootstrap required, opened a new log
I20260619 07:00:32.405861 24969 tablet_bootstrap.cc:654] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:32.405889 24970 ts_tablet_manager.cc:1397] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260619 07:00:32.406759 24969 log.cc:824] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:32.409482 24969 tablet_bootstrap.cc:492] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: No bootstrap required, opened a new log
I20260619 07:00:32.409682 24969 ts_tablet_manager.cc:1397] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 07:00:32.410063 24970 raft_consensus.cc:348] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } }
I20260619 07:00:32.410327 24970 raft_consensus.cc:374] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:32.410377 24970 raft_consensus.cc:729] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2881cd02a8264a719028641492be16ba, State: Initialized, Role: FOLLOWER
I20260619 07:00:32.410898 24970 consensus_queue.cc:260] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } }
I20260619 07:00:32.411960 24970 ts_tablet_manager.cc:1428] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260619 07:00:32.412315 24934 heartbeater.cc:503] Master 127.11.106.126:33571 was elected leader, sending a full tablet report...
I20260619 07:00:32.413493 24969 raft_consensus.cc:348] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } }
I20260619 07:00:32.413744 24969 raft_consensus.cc:374] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:32.413805 24969 raft_consensus.cc:729] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ed1f84b259439981b1d7cdb1352e90, State: Initialized, Role: FOLLOWER
I20260619 07:00:32.414672 24969 consensus_queue.cc:260] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } }
I20260619 07:00:32.415139 24965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.375072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:38747 (local address 127.0.0.1:48534)
0619 07:00:32.377237 (+ 2165us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:32.377253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:32.377352 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:32.386329 (+ 8977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:32.386333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:32.386354 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:32.386461 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.386469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.411716 (+ 25247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.411722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.413094 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.413113 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.414635 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.414640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.414754 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.414790 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 07:00:32.414851 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 07:00:32.414953 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2041,"thread_start_us":91,"threads_started":1}
I20260619 07:00:32.415925 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.375250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.11.106.65:38747)
0619 07:00:32.384838 (+ 9588us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:32.384848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:32.384877 (+ 29us) server_negotiation.cc:407] Connection header received
0619 07:00:32.385399 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:32.385406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:32.386146 (+ 740us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:32.401602 (+ 15456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:32.401804 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.403052 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.413335 (+ 10283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.414445 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.415412 (+ 967us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.415589 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:32.415651 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 07:00:32.415757 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9442,"thread_start_us":87,"threads_started":1}
I20260619 07:00:32.419322 24594 tablet_service.cc:1467] Processing CreateTablet for tablet 015a22a6e3e04344b72304165bc490ed (DEFAULT_TABLE table=test-table [id=edf1e7c90f7644229c10a90946f58214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:32.420472 24594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015a22a6e3e04344b72304165bc490ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:32.420684 24969 ts_tablet_manager.cc:1428] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260619 07:00:32.421010 24800 heartbeater.cc:503] Master 127.11.106.126:33571 was elected leader, sending a full tablet report...
I20260619 07:00:32.431063 24975 tablet_bootstrap.cc:492] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0: Bootstrap starting.
I20260619 07:00:32.433794 24975 tablet_bootstrap.cc:654] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:32.434723 24975 log.cc:824] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:32.437278 24975 tablet_bootstrap.cc:492] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0: No bootstrap required, opened a new log
I20260619 07:00:32.437471 24975 ts_tablet_manager.cc:1397] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 07:00:32.441790 24975 raft_consensus.cc:348] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } }
I20260619 07:00:32.442165 24975 raft_consensus.cc:374] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:32.442284 24975 raft_consensus.cc:729] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46eea7a24dad450fb3763f33b709fba0, State: Initialized, Role: FOLLOWER
I20260619 07:00:32.442906 24975 consensus_queue.cc:260] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } }
I20260619 07:00:32.444096 24975 ts_tablet_manager.cc:1428] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 07:00:32.444591 24666 heartbeater.cc:503] Master 127.11.106.126:33571 was elected leader, sending a full tablet report...
W20260619 07:00:32.512830 24667 tablet.cc:2369] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 07:00:32.522395 24801 tablet.cc:2369] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 07:00:32.546612 24935 tablet.cc:2369] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:00:32.748916 24977 raft_consensus.cc:482] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:32.749189 24977 raft_consensus.cc:504] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } }
I20260619 07:00:32.750633 24977 leader_election.cc:288] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2881cd02a8264a719028641492be16ba (127.11.106.67:40997), 78ed1f84b259439981b1d7cdb1352e90 (127.11.106.66:38301)
I20260619 07:00:32.755831 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.751512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:34721 (local address 127.11.106.66:38301)
0619 07:00:32.751641 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:32.751646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:32.751842 (+ 196us) server_negotiation.cc:407] Connection header received
0619 07:00:32.752019 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:32.752023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:32.752073 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:32.752131 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:32.752754 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.753358 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.754788 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.755194 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.755345 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.755408 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:32.755610 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 07:00:32.755688 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 07:00:32.755860 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.750956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:38301 (local address 127.11.106.65:34721)
0619 07:00:32.751737 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:32.751755 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:32.751879 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:32.752317 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:32.752322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:32.752342 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:32.752593 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.752605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.753525 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.753528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.754682 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.754695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.755337 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.755341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.755473 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.755495 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 07:00:32.755595 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 07:00:32.755677 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":3,"thread_start_us":127,"threads_started":1}
I20260619 07:00:32.756024 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.751177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:36745 (local address 127.11.106.67:40997)
0619 07:00:32.751317 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:32.751322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:32.751527 (+ 205us) server_negotiation.cc:407] Connection header received
0619 07:00:32.751751 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:32.751755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:32.751808 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:32.751896 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:32.752256 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.753104 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.754733 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.755335 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.755449 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.755497 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:32.755804 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 07:00:32.755884 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 07:00:32.756410 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.750956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:40997 (local address 127.11.106.65:36745)
0619 07:00:32.751408 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:32.751425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:32.751628 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:32.751920 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:32.751926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:32.751945 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:32.752128 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.752136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.753241 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.753245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.754586 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:32.754599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.755561 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.755565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:32.755671 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.755694 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 07:00:32.756185 (+ 491us) client_negotiation.cc:240] Negotiation successful
0619 07:00:32.756259 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":4,"thread_start_us":155,"threads_started":1}
I20260619 07:00:32.756767 24750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015a22a6e3e04344b72304165bc490ed" candidate_uuid: "46eea7a24dad450fb3763f33b709fba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ed1f84b259439981b1d7cdb1352e90" is_pre_election: true
I20260619 07:00:32.757124 24750 raft_consensus.cc:2454] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46eea7a24dad450fb3763f33b709fba0 in term 0.
I20260619 07:00:32.757174 24881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015a22a6e3e04344b72304165bc490ed" candidate_uuid: "46eea7a24dad450fb3763f33b709fba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2881cd02a8264a719028641492be16ba" is_pre_election: true
I20260619 07:00:32.757442 24881 raft_consensus.cc:2454] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46eea7a24dad450fb3763f33b709fba0 in term 0.
I20260619 07:00:32.757668 24554 leader_election.cc:302] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46eea7a24dad450fb3763f33b709fba0, 78ed1f84b259439981b1d7cdb1352e90; no voters:
I20260619 07:00:32.757994 24977 raft_consensus.cc:2781] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:00:32.758090 24977 raft_consensus.cc:482] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:00:32.758158 24977 raft_consensus.cc:3037] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:32.759479 24977 raft_consensus.cc:504] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } }
I20260619 07:00:32.759852 24977 leader_election.cc:288] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [CANDIDATE]: Term 1 election: Requested vote from peers 2881cd02a8264a719028641492be16ba (127.11.106.67:40997), 78ed1f84b259439981b1d7cdb1352e90 (127.11.106.66:38301)
I20260619 07:00:32.760176 24881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015a22a6e3e04344b72304165bc490ed" candidate_uuid: "46eea7a24dad450fb3763f33b709fba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2881cd02a8264a719028641492be16ba"
I20260619 07:00:32.760293 24881 raft_consensus.cc:3037] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:32.760425 24750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015a22a6e3e04344b72304165bc490ed" candidate_uuid: "46eea7a24dad450fb3763f33b709fba0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ed1f84b259439981b1d7cdb1352e90"
I20260619 07:00:32.760582 24750 raft_consensus.cc:3037] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:32.761520 24881 raft_consensus.cc:2454] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46eea7a24dad450fb3763f33b709fba0 in term 1.
I20260619 07:00:32.761857 24553 leader_election.cc:302] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2881cd02a8264a719028641492be16ba, 46eea7a24dad450fb3763f33b709fba0; no voters:
I20260619 07:00:32.761894 24750 raft_consensus.cc:2454] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46eea7a24dad450fb3763f33b709fba0 in term 1.
I20260619 07:00:32.762096 24977 raft_consensus.cc:2781] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:00:32.762313 24977 raft_consensus.cc:686] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [term 1 LEADER]: Becoming Leader. State: Replica: 46eea7a24dad450fb3763f33b709fba0, State: Running, Role: LEADER
I20260619 07:00:32.762636 24977 consensus_queue.cc:237] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } }
I20260619 07:00:32.766036 24467 catalog_manager.cc:5495] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46eea7a24dad450fb3763f33b709fba0 (127.11.106.65). New cstate: current_term: 1 leader_uuid: "46eea7a24dad450fb3763f33b709fba0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46eea7a24dad450fb3763f33b709fba0" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38747 } health_report { overall_health: HEALTHY } } }
I20260619 07:00:32.830278 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.826571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.11.106.65:38747)
0619 07:00:32.826725 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:32.826728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:32.826778 (+ 50us) server_negotiation.cc:407] Connection header received
0619 07:00:32.826904 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:32.826906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:32.826960 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:32.827053 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:00:32.827279 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.827886 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.828889 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:32.829088 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:32.829162 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:32.829504 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:00:32.829930 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:00:32.830017 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:32.830115 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 07:00:32.830163 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 07:00:32.835826 24881 raft_consensus.cc:1261] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [term 1 FOLLOWER]: Refusing update from remote peer 46eea7a24dad450fb3763f33b709fba0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:00:32.835979 24750 raft_consensus.cc:1261] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [term 1 FOLLOWER]: Refusing update from remote peer 46eea7a24dad450fb3763f33b709fba0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:00:32.836571 24977 consensus_queue.cc:1035] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2881cd02a8264a719028641492be16ba" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 40997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:00:32.836748 24987 consensus_queue.cc:1035] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78ed1f84b259439981b1d7cdb1352e90" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:00:32.843194 24990 mvcc.cc:205] Tried to move back new op lower bound from 7298467564888707072 to 7298467564598730752. Current Snapshot: MvccSnapshot[applied={T|T < 7298467564888707072}]
I20260619 07:00:32.846655 24991 mvcc.cc:205] Tried to move back new op lower bound from 7298467564888707072 to 7298467564598730752. Current Snapshot: MvccSnapshot[applied={T|T < 7298467564888707072}]
I20260619 07:00:32.852176 24467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:00:32.852346 24467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:00:32.854960 24467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d339b48de74b44c9a1361b6c8935af34: Sending DeleteTablet for 3 replicas of tablet 015a22a6e3e04344b72304165bc490ed
I20260619 07:00:32.855762 24867 tablet_service.cc:1514] Processing DeleteTablet for tablet 015a22a6e3e04344b72304165bc490ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:32 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:42800
I20260619 07:00:32.855911 24733 tablet_service.cc:1514] Processing DeleteTablet for tablet 015a22a6e3e04344b72304165bc490ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:32 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:58812
I20260619 07:00:32.856262 24594 tablet_service.cc:1514] Processing DeleteTablet for tablet 015a22a6e3e04344b72304165bc490ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:32 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:48534
I20260619 07:00:32.856707 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 24506
I20260619 07:00:32.856755 24999 tablet_replica.cc:330] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: stopping tablet replica
I20260619 07:00:32.856948 25001 tablet_replica.cc:330] T 015a22a6e3e04344b72304165bc490ed P 46eea7a24dad450fb3763f33b709fba0: stopping tablet replica
I20260619 07:00:32.857609 24999 raft_consensus.cc:2229] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:00:32.857934 24999 raft_consensus.cc:2258] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:00:32.858753 25000 tablet_replica.cc:330] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: stopping tablet replica
I20260619 07:00:32.859026 25000 raft_consensus.cc:2229] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:00:32.859329 25000 raft_consensus.cc:2258] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:00:32.860261 24999 ts_tablet_manager.cc:1905] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:00:32.860744 25000 ts_tablet_manager.cc:1905] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:00:32.862924 25000 ts_tablet_manager.cc:1918] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:00:32.863037 25000 log.cc:1196] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/015a22a6e3e04344b72304165bc490ed
I20260619 07:00:32.863329 25000 ts_tablet_manager.cc:1939] T 015a22a6e3e04344b72304165bc490ed P 78ed1f84b259439981b1d7cdb1352e90: Deleting consensus metadata
I20260619 07:00:32.863690 24999 ts_tablet_manager.cc:1918] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:00:32.863793 24999 log.cc:1196] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/015a22a6e3e04344b72304165bc490ed
I20260619 07:00:32.864109 24999 ts_tablet_manager.cc:1939] T 015a22a6e3e04344b72304165bc490ed P 2881cd02a8264a719028641492be16ba: Deleting consensus metadata
I20260619 07:00:32.864589 24453 catalog_manager.cc:4841] TS 78ed1f84b259439981b1d7cdb1352e90 (127.11.106.66:38301): tablet 015a22a6e3e04344b72304165bc490ed (table test-table [id=edf1e7c90f7644229c10a90946f58214]) successfully deleted
W20260619 07:00:32.864861 24453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:38747 (error 108)
I20260619 07:00:32.864984 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 24670
I20260619 07:00:32.866145 24452 catalog_manager.cc:4841] TS 2881cd02a8264a719028641492be16ba (127.11.106.67:40997): tablet 015a22a6e3e04344b72304165bc490ed (table test-table [id=edf1e7c90f7644229c10a90946f58214]) successfully deleted
I20260619 07:00:32.866974 24965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:32.866709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:38747 (local address 127.0.0.1:39306)
0619 07:00:32.866846 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:32.866906 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 07:00:32.867144 24453 catalog_manager.cc:4584] TS 46eea7a24dad450fb3763f33b709fba0 (127.11.106.65:38747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 015a22a6e3e04344b72304165bc490ed: Network error: Client connection negotiation failed: client connection to 127.11.106.65:38747: connect: Connection refused (error 111)
I20260619 07:00:32.870904 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 24803
I20260619 07:00:32.876398 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 24435
I20260619 07:00:32.883134 11689 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 07:00:33.228165 11689 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 07:00:33.228297 11689 mini_ranger.cc:66] Stopping Ranger...
I20260619 07:00:33.573199 11689 mini_ranger.cc:68] Stopped Ranger
2026-06-19 07:00:33.573 UTC [23502] LOG: received smart shutdown request
2026-06-19 07:00:33.577 UTC [23502] LOG: background worker "logical replication launcher" (PID 23511) exited with exit code 1
2026-06-19 07:00:33.577 UTC [23506] LOG: shutting down
2026-06-19 07:00:33.744 UTC [23502] LOG: database system is shut down
2026-06-19T07:00:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49264 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:00:33 dist-test-slave-10d0 krb5kdc[25011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:00:33 dist-test-slave-10d0 krb5kdc[25011](info): set up 2 sockets
Jun 19 07:00:33 dist-test-slave-10d0 krb5kdc[25011](info): commencing operation
krb5kdc: starting...
W20260619 07:00:34.905196 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:00:34 dist-test-slave-10d0 krb5kdc[25011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:00:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:00:35.024471 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:35445
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:35445,127.11.106.125:45531,127.11.106.124:36473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:35.135581 25024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:35.135830 25024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:35.135870 25024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:35.139951 25024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:35.140012 25024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:35.140033 25024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:35.140049 25024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:35.140064 25024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:35.145316 25024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:35445,127.11.106.125:45531,127.11.106.124:36473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:35445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:35.146355 25024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:35.147389 25024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:35.156002 25032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:35.156038 25030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:35.156019 25029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:35.156189 25024 server_base.cc:1034] running on GCE node
I20260619 07:00:35.156939 25024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:35.157974 25024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:35.159179 25024 hybrid_clock.cc:648] HybridClock initialized: now 1781852435159164 us; error 37 us; skew 500 ppm
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:35.162263 25024 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:00:35.163185 25024 webserver.cc:460] Webserver started at http://127.11.106.126:34835/ using document root <none> and password file <none>
I20260619 07:00:35.163740 25024 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:35.163872 25024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:35.164117 25024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:35.166177 25024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "651df1cfe5674ae8aab647fa7ba3a5bd"
format_stamp: "Formatted at 2026-06-19 07:00:35 on dist-test-slave-10d0"
server_key: "aaed65e97078e684ac6d530592f68166"
server_key_iv: "89ec872cd9a055abdf0afa7b1afa99a0"
server_key_version: "encryptionkey@0"
I20260619 07:00:35.166728 25024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "651df1cfe5674ae8aab647fa7ba3a5bd"
format_stamp: "Formatted at 2026-06-19 07:00:35 on dist-test-slave-10d0"
server_key: "aaed65e97078e684ac6d530592f68166"
server_key_iv: "89ec872cd9a055abdf0afa7b1afa99a0"
server_key_version: "encryptionkey@0"
I20260619 07:00:35.170627 25024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 07:00:35.173074 25039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:35.174369 25024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:00:35.174508 25024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "651df1cfe5674ae8aab647fa7ba3a5bd"
format_stamp: "Formatted at 2026-06-19 07:00:35 on dist-test-slave-10d0"
server_key: "aaed65e97078e684ac6d530592f68166"
server_key_iv: "89ec872cd9a055abdf0afa7b1afa99a0"
server_key_version: "encryptionkey@0"
I20260619 07:00:35.174618 25024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:35.235551 25024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:35.238986 25024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:35.239195 25024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:35.246635 25024 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:35445
I20260619 07:00:35.246939 25091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:35445 every 8 connection(s)
I20260619 07:00:35.247613 25024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:00:35.250331 25092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:35.251202 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 25024
I20260619 07:00:35.251300 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:00:35.251600 11689 external_mini_cluster.cc:1428] Setting key 80c74fc35a52ccae8647792fb8dcab4c
I20260619 07:00:35.254204 25092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.106.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.125@KRBTEST.COM" created.
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:35.270267 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.256567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:35445 (local address 127.0.0.1:33918)
0619 07:00:35.257216 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.257256 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:35.258194 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:35.258949 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:35.258955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:35.259421 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:35.259672 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.259686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.261031 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.261034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.261868 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.261878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.262587 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.262591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.262717 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.263341 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:35.263358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:35.264907 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:35.267294 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.267303 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.267313 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.267574 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.267952 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.267954 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.267955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.268061 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.268853 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:35.268859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:35.268997 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 07:00:35.269152 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 07:00:35.269412 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":146,"threads_started":1}
I20260619 07:00:35.270284 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.256582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.11.106.126:35445)
0619 07:00:35.257026 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.257048 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.257372 (+ 324us) server_negotiation.cc:407] Connection header received
0619 07:00:35.258554 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.258559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.258745 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.258949 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.259833 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.260879 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.261998 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.262390 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.262605 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.265065 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.265075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.265091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.265123 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.267154 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.267695 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.267704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.267707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.267848 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.268156 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.268162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.268165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.268714 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.268866 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.269254 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.269523 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
I20260619 07:00:35.272887 25092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:00:35.273597 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.273365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:45531 (local address 127.0.0.1:52794)
0619 07:00:35.273470 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.273534 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 07:00:35.273805 25041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
I20260619 07:00:35.274612 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.274400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:45531 (local address 127.0.0.1:52802)
0619 07:00:35.274507 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.274561 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 07:00:35.274834 25092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.125:45531: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab.
Entry for principal kudu/127.11.106.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab.
I20260619 07:00:35.277727 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.125:45531
--webserver_interface=127.11.106.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab
--principal=kudu/127.11.106.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:35445,127.11.106.125:45531,127.11.106.124:36473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:00:35.324133 25092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } attempt: 1
I20260619 07:00:35.325104 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.324785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:45531 (local address 127.0.0.1:52806)
0619 07:00:35.324945 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.325027 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 07:00:35.325989 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.325738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:45531 (local address 127.0.0.1:52812)
0619 07:00:35.325873 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.325929 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 07:00:35.326133 25092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.125:45531: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
W20260619 07:00:35.387696 25099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:35.387992 25099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:35.388075 25099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:35.392081 25099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:35.392168 25099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:35.392210 25099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:35.392244 25099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:35.392278 25099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:35.394304 25092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } attempt: 2
I20260619 07:00:35.395138 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.394848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:45531 (local address 127.0.0.1:52824)
0619 07:00:35.394995 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.395060 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 07:00:35.395874 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.395651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:45531 (local address 127.0.0.1:52832)
0619 07:00:35.395762 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.395818 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 07:00:35.396003 25092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.125:45531: Network error: Client connection negotiation failed: client connection to 127.11.106.125:45531: connect: Connection refused (error 111)
I20260619 07:00:35.398025 25099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:35445,127.11.106.125:45531,127.11.106.124:36473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab
--principal=kudu/127.11.106.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.125:45531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.106.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:35.399066 25099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:35.400087 25099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:35.408591 25104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:35.408617 25107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:35.408728 25099 server_base.cc:1034] running on GCE node
W20260619 07:00:35.408983 25105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:35.409490 25099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:35.410616 25099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:35.411804 25099 hybrid_clock.cc:648] HybridClock initialized: now 1781852435411780 us; error 45 us; skew 500 ppm
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:35.414499 25099 init.cc:377] Logged in from keytab as kudu/127.11.106.125@KRBTEST.COM (short username kudu)
I20260619 07:00:35.415382 25099 webserver.cc:460] Webserver started at http://127.11.106.125:34301/ using document root <none> and password file <none>
I20260619 07:00:35.415921 25099 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:35.416010 25099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:35.416236 25099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:35.418265 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/data/instance:
uuid: "0a62143be7684975a3d179a39444a380"
format_stamp: "Formatted at 2026-06-19 07:00:35 on dist-test-slave-10d0"
server_key: "7167e209ac765eb5a180c145cb108a14"
server_key_iv: "a8842c576bcbaa25d2874ea938bd06d0"
server_key_version: "encryptionkey@0"
I20260619 07:00:35.418821 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/wal/instance:
uuid: "0a62143be7684975a3d179a39444a380"
format_stamp: "Formatted at 2026-06-19 07:00:35 on dist-test-slave-10d0"
server_key: "7167e209ac765eb5a180c145cb108a14"
server_key_iv: "a8842c576bcbaa25d2874ea938bd06d0"
server_key_version: "encryptionkey@0"
I20260619 07:00:35.422336 25099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 07:00:35.424708 25114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:35.426213 25099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 07:00:35.426419 25099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/wal
uuid: "0a62143be7684975a3d179a39444a380"
format_stamp: "Formatted at 2026-06-19 07:00:35 on dist-test-slave-10d0"
server_key: "7167e209ac765eb5a180c145cb108a14"
server_key_iv: "a8842c576bcbaa25d2874ea938bd06d0"
server_key_version: "encryptionkey@0"
I20260619 07:00:35.426611 25099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:35.437042 25099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:35.440140 25099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:35.440330 25099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:35.446955 25099 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.125:45531
I20260619 07:00:35.447245 25166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.125:45531 every 8 connection(s)
I20260619 07:00:35.447891 25099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
I20260619 07:00:35.450513 25167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:35.454213 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 25099
I20260619 07:00:35.454056 25167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:00:35.454356 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-1/wal/instance
I20260619 07:00:35.454646 11689 external_mini_cluster.cc:1428] Setting key 5b4dc823865c749f8baaeb6fe13aa03e
WARNING: no policy specified for kudu/127.11.106.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.124@KRBTEST.COM" created.
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:35.470247 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.455976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.11.106.126:35445)
0619 07:00:35.456132 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.456139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.457168 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 07:00:35.458127 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.458131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.458191 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.458310 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.459497 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.460035 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.461214 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.461458 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.461571 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.466272 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.466275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.466277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.466304 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.467989 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.468491 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.468498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.468502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.468620 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.468959 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.468963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.468967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.469238 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.469352 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.469852 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.470008 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 07:00:35.470826 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.456415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:35445 (local address 127.0.0.1:33932)
0619 07:00:35.456980 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.457038 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:35.457917 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:35.458478 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:35.458485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:35.458880 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:35.459347 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.459359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.460175 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.460178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.461081 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.461091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.461606 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.461611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.461782 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.462373 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:35.462388 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:35.466054 (+ 3666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:35.468103 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.468108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.468119 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.468371 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.468716 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.468718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.468720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.468846 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.469346 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:35.469350 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:35.469576 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 07:00:35.469728 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 07:00:35.469964 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":168,"threads_started":1}
I20260619 07:00:35.472296 25167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab.
Entry for principal kudu/127.11.106.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab.
I20260619 07:00:35.481990 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.124:36473
--webserver_interface=127.11.106.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab
--principal=kudu/127.11.106.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:35445,127.11.106.125:45531,127.11.106.124:36473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:00:35.482484 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.472869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:45531 (local address 127.0.0.1:52844)
0619 07:00:35.472961 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.472974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:35.473044 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:35.473710 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:35.473712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:35.473834 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:35.473913 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.473918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.474914 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.474916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.475519 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.475528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.476158 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.476161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.476241 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.476753 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:35.476764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:35.478253 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:35.480609 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.480615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.480620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.480909 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.481252 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.481259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.481264 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.481363 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.481971 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:35.481973 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:35.482065 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 07:00:35.482229 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 07:00:35.482358 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 07:00:35.483007 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.473011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.11.106.125:45531)
0619 07:00:35.473349 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.473363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.473389 (+ 26us) server_negotiation.cc:407] Connection header received
0619 07:00:35.473437 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.473442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.473591 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.473696 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.474065 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.474813 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.475651 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.476052 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.476144 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.478402 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.478415 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.478434 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.478455 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.480439 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.481044 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.481050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.481052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.481142 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.481457 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.481460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.481462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.481839 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.481993 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.482753 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.482867 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20260619 07:00:35.485469 25167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:00:35.486233 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.485956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43042)
0619 07:00:35.486088 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.486178 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 07:00:35.486483 25117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
I20260619 07:00:35.487185 25092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } attempt: 3
I20260619 07:00:35.487350 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.487094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43046)
0619 07:00:35.487222 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.487271 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 07:00:35.487798 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36473: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
I20260619 07:00:35.496989 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.487631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:45531 (local address 127.0.0.1:52846)
0619 07:00:35.487794 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.487813 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:35.487908 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:35.488170 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:35.488174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:35.488381 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:35.488534 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.488575 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.489301 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.489308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.490313 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.490329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.490957 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.490962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.491096 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.491552 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:35.491574 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:35.493691 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:35.495466 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.495468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.495470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.495660 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.495924 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.495926 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.495928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.496014 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.496466 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:35.496469 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:35.496666 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 07:00:35.496767 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 07:00:35.496875 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 07:00:35.497078 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.487691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.11.106.125:45531)
0619 07:00:35.487946 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.487952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.487967 (+ 15us) server_negotiation.cc:407] Connection header received
0619 07:00:35.488012 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.488014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.488058 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.488119 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.488687 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.489189 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.490543 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.490817 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.490920 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.493838 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.493842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.493844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.493867 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.495355 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.495749 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.495752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.495754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.495832 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.496137 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.496140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.496142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.496343 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.496436 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.496830 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.496940 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260619 07:00:35.498055 25092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:00:35.498770 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.498498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43062)
0619 07:00:35.498643 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.498704 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 07:00:35.499454 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.499213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43074)
0619 07:00:35.499354 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.499400 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 07:00:35.499680 25092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36473: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
I20260619 07:00:35.530872 25092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } attempt: 1
I20260619 07:00:35.531971 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.531583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43084)
0619 07:00:35.531776 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.531843 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 07:00:35.532809 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.532524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43088)
0619 07:00:35.532686 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.532739 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 07:00:35.532966 25092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36473: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
I20260619 07:00:35.537050 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } attempt: 1
I20260619 07:00:35.537782 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.537491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43090)
0619 07:00:35.537656 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.537711 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 07:00:35.538515 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.538313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43106)
0619 07:00:35.538415 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.538459 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 07:00:35.538738 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36473: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
W20260619 07:00:35.595796 25173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:35.596057 25173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:35.596097 25173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:35.600071 25173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:35.600126 25173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:35.600147 25173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:35.600167 25173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:35.600220 25173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:35.605429 25173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:35445,127.11.106.125:45531,127.11.106.124:36473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab
--principal=kudu/127.11.106.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.124:36473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.106.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:35.606492 25173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:35.606914 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } attempt: 2
I20260619 07:00:35.607739 25173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 07:00:35.607862 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.607492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43110)
0619 07:00:35.607683 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.607746 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 07:00:35.608124 25092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } attempt: 2
I20260619 07:00:35.608779 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.608427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43120)
0619 07:00:35.608653 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.608704 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260619 07:00:35.608918 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36473: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
I20260619 07:00:35.608974 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.608597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43134)
0619 07:00:35.608753 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.608810 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 07:00:35.609612 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.609407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43148)
0619 07:00:35.609508 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.609555 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 07:00:35.609767 25092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36473: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36473: connect: Connection refused (error 111)
W20260619 07:00:35.616600 25179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:35.616510 25180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:35.616729 25173 server_base.cc:1034] running on GCE node
W20260619 07:00:35.616503 25182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:35.617381 25173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:35.618454 25173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:35.619625 25173 hybrid_clock.cc:648] HybridClock initialized: now 1781852435619606 us; error 29 us; skew 500 ppm
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:35.622394 25173 init.cc:377] Logged in from keytab as kudu/127.11.106.124@KRBTEST.COM (short username kudu)
I20260619 07:00:35.623217 25173 webserver.cc:460] Webserver started at http://127.11.106.124:35495/ using document root <none> and password file <none>
I20260619 07:00:35.623764 25173 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:35.623855 25173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:35.624094 25173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:35.626142 25173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/data/instance:
uuid: "90f81baea69346558019201f49224c55"
format_stamp: "Formatted at 2026-06-19 07:00:35 on dist-test-slave-10d0"
server_key: "2c687962420ec57a495235f448d5458f"
server_key_iv: "4364b772600cd0314ff92015c593efd2"
server_key_version: "encryptionkey@0"
I20260619 07:00:35.626761 25173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/wal/instance:
uuid: "90f81baea69346558019201f49224c55"
format_stamp: "Formatted at 2026-06-19 07:00:35 on dist-test-slave-10d0"
server_key: "2c687962420ec57a495235f448d5458f"
server_key_iv: "4364b772600cd0314ff92015c593efd2"
server_key_version: "encryptionkey@0"
I20260619 07:00:35.630407 25173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 07:00:35.632901 25189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:35.634090 25173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:00:35.634227 25173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/wal
uuid: "90f81baea69346558019201f49224c55"
format_stamp: "Formatted at 2026-06-19 07:00:35 on dist-test-slave-10d0"
server_key: "2c687962420ec57a495235f448d5458f"
server_key_iv: "4364b772600cd0314ff92015c593efd2"
server_key_version: "encryptionkey@0"
I20260619 07:00:35.634331 25173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:35.658921 25173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:35.662330 25173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:35.662487 25173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:35.669499 25173 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.124:36473
I20260619 07:00:35.669790 25241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.124:36473 every 8 connection(s)
I20260619 07:00:35.670470 25173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
I20260619 07:00:35.673310 25242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:35.676930 25242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:00:35.678851 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 25173
I20260619 07:00:35.678960 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781852273483449-11689-0/minicluster-data/master-2/wal/instance
I20260619 07:00:35.679253 11689 external_mini_cluster.cc:1428] Setting key 064253486824ef5063781fde62ff6fa5
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
I20260619 07:00:35.690395 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.678776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.11.106.126:35445)
0619 07:00:35.678914 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.678919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.679958 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 07:00:35.681025 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.681028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.681085 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.681208 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.682061 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.682567 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.683529 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.683753 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.683862 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.686444 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.686448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.686451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.686484 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.688238 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.688848 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.688855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.688857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.688949 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.689300 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.689304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.689305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.689518 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.689626 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.690107 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.690226 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 07:00:35.691236 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.679232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:35445 (local address 127.0.0.1:33944)
0619 07:00:35.679780 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.679835 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:35.680829 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:35.681254 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:35.681260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:35.681587 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:35.681922 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.681935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.682682 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.682686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.683399 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.683407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.683896 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.683900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.684102 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.684714 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:35.684736 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:35.686255 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:35.688355 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.688362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.688369 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.688754 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.689057 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.689060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.689062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.689194 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.689635 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:35.689644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:35.689877 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 07:00:35.690049 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 07:00:35.690346 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":57,"thread_start_us":170,"threads_started":1}
I20260619 07:00:35.691237 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.679884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.11.106.124:36473)
0619 07:00:35.680513 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.680521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.680604 (+ 83us) server_negotiation.cc:407] Connection header received
0619 07:00:35.680856 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.680883 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.681078 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.681289 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.682174 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.682989 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.683816 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.684202 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.684320 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.686890 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.686898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.686918 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.686961 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.688810 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.689210 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.689216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.689218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.689300 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.689578 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.689581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.689583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.689870 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.690065 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.690292 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.690427 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":549,"thread_start_us":142,"threads_started":1}
I20260619 07:00:35.692607 25242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:35.700067 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } attempt: 3
I20260619 07:00:35.700963 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.693248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.11.106.125:45531)
0619 07:00:35.693403 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.693408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.693423 (+ 15us) server_negotiation.cc:407] Connection header received
0619 07:00:35.693466 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.693469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.693514 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.693600 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.694055 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.694558 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.695320 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.695574 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.695685 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.697611 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.697621 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.697623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.697644 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.699133 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.699603 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.699606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.699608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.699696 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.699995 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.699998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.700000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.700257 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.700356 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.700694 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.700807 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 07:00:35.701363 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.693123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:45531 (local address 127.0.0.1:52850)
0619 07:00:35.693293 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.693307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:35.693381 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:35.693641 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:35.693647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:35.693815 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:35.693927 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.693934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.694662 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.694668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.695205 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.695212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.695693 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.695696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.695800 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.696223 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:35.696237 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:35.697480 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:35.699249 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.699253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.699256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.699509 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.699796 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.699800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.699803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.699898 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.700350 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:35.700356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:35.700455 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 07:00:35.701097 (+ 642us) client_negotiation.cc:240] Negotiation successful
0619 07:00:35.701216 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260619 07:00:35.702191 25242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:00:35.702589 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.693867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.11.106.126:35445)
0619 07:00:35.693979 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.693983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.693995 (+ 12us) server_negotiation.cc:407] Connection header received
0619 07:00:35.694090 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.694093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.694128 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.694230 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.694779 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.695261 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.696034 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.696391 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.696578 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.698942 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.698947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.698951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.698978 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.700734 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.701181 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.701186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.701193 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.701280 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.701733 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.701736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.701737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.701952 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.702165 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.702303 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.702416 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
I20260619 07:00:35.708921 25092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } attempt: 3
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
I20260619 07:00:35.710570 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.700601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.11.106.124:36473)
0619 07:00:35.700879 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.700883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.700894 (+ 11us) server_negotiation.cc:407] Connection header received
0619 07:00:35.700924 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.700926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.700959 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.701272 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.701850 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.702491 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.703859 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.704243 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.704335 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.706327 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.706330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.706332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.706351 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.708422 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.708980 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.708984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.708985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.709063 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.709386 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.709390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.709392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.709602 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.710208 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.710299 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.710413 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260619 07:00:35.710572 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.700473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43168)
0619 07:00:35.700667 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.700681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:35.700742 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:35.701309 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:35.701315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:35.701488 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:35.701592 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.701600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.702628 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.702633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.703452 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.703462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.704349 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.704353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.704437 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.704900 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:35.704913 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:35.706206 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:35.708600 (+ 2394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.708605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.708607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.708859 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.709165 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.709169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.709171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.709267 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.709732 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:35.709735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:35.709833 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 07:00:35.709915 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 07:00:35.710058 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 07:00:35.713965 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.703766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.11.106.125:45531)
0619 07:00:35.703859 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.703862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.703905 (+ 43us) server_negotiation.cc:407] Connection header received
0619 07:00:35.704047 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.704049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.704091 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.704166 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.705345 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.705796 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.706841 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.707190 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.707302 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.710113 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.710118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.710122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.710148 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.712030 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.712529 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.712532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.712534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.712630 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.712934 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.712938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.712939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.713142 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.713296 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.713678 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.713797 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 07:00:35.714161 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.702628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43172)
0619 07:00:35.702810 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.702823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:35.702891 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:35.703346 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:35.703349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:35.703508 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:35.703604 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.703610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.705105 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.705111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.705865 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.705877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.706791 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.706795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.706878 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.707622 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:35.707641 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:35.709691 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:35.712385 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.712388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.712390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.712631 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.712913 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.712919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.712921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.713013 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.713446 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:35.713449 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:35.713524 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 07:00:35.713877 (+ 353us) client_negotiation.cc:240] Negotiation successful
0619 07:00:35.714012 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260619 07:00:35.714175 25167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380: Bootstrap starting.
I20260619 07:00:35.714700 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.702795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.11.106.124:36473)
0619 07:00:35.703115 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.703121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.703131 (+ 10us) server_negotiation.cc:407] Connection header received
0619 07:00:35.703179 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.703182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.703225 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.703310 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.703712 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.705013 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.706334 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.706689 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.706802 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.709820 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.709823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.709825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.709849 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.712279 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.712732 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.712736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.712739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.712815 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.713105 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.713107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.713109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.713329 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.713416 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.714475 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.714587 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
Jun 19 07:00:35 dist-test-slave-10d0 krb5kdc[25011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852435, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
I20260619 07:00:35.717314 25167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:35.717639 25242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55: Bootstrap starting.
I20260619 07:00:35.718250 25167 log.cc:824] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:35.719383 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.709328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36473 (local address 127.0.0.1:43174)
0619 07:00:35.709508 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:35.709522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:35.709600 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:35.709992 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:35.709994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:35.710176 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:35.710297 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.710305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.711943 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.711947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.712749 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:35.712761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.713355 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.713359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:35.713436 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.713873 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:35.713887 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:35.715201 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:35.717835 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.717839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.717841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.718099 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.718415 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:35.718418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:35.718419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:35.718523 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:35.718898 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:35.718901 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:35.718999 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 07:00:35.719109 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 07:00:35.719238 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 07:00:35.719439 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.709362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.11.106.124:36473)
0619 07:00:35.709747 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.709760 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.709771 (+ 11us) server_negotiation.cc:407] Connection header received
0619 07:00:35.709801 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.709804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.709851 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.710404 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.710519 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.711809 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.712894 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.713219 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.713632 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.715309 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.715312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.715315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.715337 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.717683 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.718220 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.718223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.718225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.718312 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.718627 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.718631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.718632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.718803 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.718878 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.719191 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.719306 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":121,"threads_started":1}
I20260619 07:00:35.720135 25242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:35.720893 25167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380: No bootstrap required, opened a new log
I20260619 07:00:35.720944 25242 log.cc:824] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:35.722321 25092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd: Bootstrap starting.
I20260619 07:00:35.722839 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:00:35.723532 25242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55: No bootstrap required, opened a new log
I20260619 07:00:35.724797 25092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:35.725054 25167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } }
I20260619 07:00:35.725397 25167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:35.725466 25167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a62143be7684975a3d179a39444a380, State: Initialized, Role: FOLLOWER
I20260619 07:00:35.725589 25092 log.cc:824] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:35.725991 25167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } }
I20260619 07:00:35.727734 25092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd: No bootstrap required, opened a new log
I20260619 07:00:35.727787 25167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:35.728104 25242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } }
I20260619 07:00:35.728310 25242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:35.728355 25242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f81baea69346558019201f49224c55, State: Initialized, Role: FOLLOWER
I20260619 07:00:35.728826 25242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } }
I20260619 07:00:35.730141 25242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:35.732491 25265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.733458 25265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:00:35.733600 25259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.733819 25259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:00:35.733800 25173 master_runner.cc:186] Error getting master registration for 127.11.106.126:35445: OK, instance_id { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" instance_seqno: 1781852435244703 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 07:00:35.737521 25173 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 07:00:35.739629 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.726491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.11.106.125:45531)
0619 07:00:35.726606 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.726612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.730067 (+ 3455us) server_negotiation.cc:407] Connection header received
0619 07:00:35.730233 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.730236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.730280 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.730326 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.731395 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.732149 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.734001 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.734325 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.734393 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.736025 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.736029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.736032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.736056 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.737725 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.738230 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.738234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.738236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.738316 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.738656 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.738661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.738662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.738879 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.739025 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.739348 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.739465 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 07:00:35.731719 25099 master_runner.cc:186] Error getting master registration for 127.11.106.126:35445: OK, instance_id { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" instance_seqno: 1781852435244703 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 07:00:35.740173 25099 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 07:00:35.740442 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.725242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.11.106.124:36473)
0619 07:00:35.725344 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.725349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.726925 (+ 1576us) server_negotiation.cc:407] Connection header received
0619 07:00:35.727097 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.727100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.727143 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.727203 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.728160 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.729012 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.731535 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.731903 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.731978 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.735681 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.735687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.735691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.735718 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.738091 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.738898 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.738903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.738906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.739002 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.739331 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.739337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.739342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.739555 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.739697 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.740140 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.740275 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 07:00:35.740782 25092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } }
I20260619 07:00:35.741077 25092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:35.741142 25092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651df1cfe5674ae8aab647fa7ba3a5bd, State: Initialized, Role: FOLLOWER
I20260619 07:00:35.741722 25092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } }
I20260619 07:00:35.742225 25283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.742386 25283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:00:35.743070 25092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:35.743649 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:35.725917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.11.106.126:35445)
0619 07:00:35.726020 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:35.726025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:35.730078 (+ 4053us) server_negotiation.cc:407] Connection header received
0619 07:00:35.730630 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:35.730634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:35.730686 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:35.730745 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:35.733805 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.734533 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.736017 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:35.736381 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:35.736473 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:35.739438 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:35.739442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:35.739444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:35.739467 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:35.741945 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.742448 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.742452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.742454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.742538 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:35.742823 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:35.742827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:35.742828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:35.742998 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:35.743120 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:35.743384 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 07:00:35.743497 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 07:00:35.747690 25282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:00:35.747839 25294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:00:35.747985 25294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 07:00:35.748208 25282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 07:00:35.750834 25271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:00:35.750910 25271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:00:35.753460 25024 master_runner.cc:263] No leader master found from master 651df1cfe5674ae8aab647fa7ba3a5bd
I20260619 07:00:35.753522 25024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 07:00:35.769326 25259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:35.769539 25259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } }
I20260619 07:00:35.770569 25259 leader_election.cc:288] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 651df1cfe5674ae8aab647fa7ba3a5bd (127.11.106.126:35445), 90f81baea69346558019201f49224c55 (127.11.106.124:36473)
I20260619 07:00:35.771427 25066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a62143be7684975a3d179a39444a380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" is_pre_election: true
I20260619 07:00:35.771759 25066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a62143be7684975a3d179a39444a380 in term 0.
I20260619 07:00:35.772403 25216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a62143be7684975a3d179a39444a380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f81baea69346558019201f49224c55" is_pre_election: true
I20260619 07:00:35.772723 25216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a62143be7684975a3d179a39444a380 in term 0.
I20260619 07:00:35.773231 25116 leader_election.cc:302] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a62143be7684975a3d179a39444a380, 651df1cfe5674ae8aab647fa7ba3a5bd; no voters:
I20260619 07:00:35.773901 25259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:00:35.773980 25259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:00:35.774021 25259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:35.775336 25259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } }
I20260619 07:00:35.775676 25259 leader_election.cc:288] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [CANDIDATE]: Term 1 election: Requested vote from peers 651df1cfe5674ae8aab647fa7ba3a5bd (127.11.106.126:35445), 90f81baea69346558019201f49224c55 (127.11.106.124:36473)
I20260619 07:00:35.776440 25216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a62143be7684975a3d179a39444a380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f81baea69346558019201f49224c55"
I20260619 07:00:35.776587 25216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:35.777839 25216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a62143be7684975a3d179a39444a380 in term 1.
I20260619 07:00:35.778252 25117 leader_election.cc:302] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a62143be7684975a3d179a39444a380, 90f81baea69346558019201f49224c55; no voters:
I20260619 07:00:35.778548 25066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a62143be7684975a3d179a39444a380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd"
I20260619 07:00:35.778693 25066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:35.780018 25066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a62143be7684975a3d179a39444a380 in term 1.
I20260619 07:00:35.782819 25259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:00:35.783099 25259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [term 1 LEADER]: Becoming Leader. State: Replica: 0a62143be7684975a3d179a39444a380, State: Running, Role: LEADER
I20260619 07:00:35.783492 25259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } }
I20260619 07:00:35.785406 25259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a62143be7684975a3d179a39444a380. Latest consensus state: current_term: 1 leader_uuid: "0a62143be7684975a3d179a39444a380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.785567 25259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:35.785976 25300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:00:35.786998 25300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:00:35.796607 25216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [term 1 FOLLOWER]: Refusing update from remote peer 0a62143be7684975a3d179a39444a380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:00:35.797437 25298 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:00:35.801800 25265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a62143be7684975a3d179a39444a380. Latest consensus state: current_term: 1 leader_uuid: "0a62143be7684975a3d179a39444a380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.801918 25265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:00:35.804790 25066 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [term 1 FOLLOWER]: Refusing update from remote peer 0a62143be7684975a3d179a39444a380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:00:35.805414 25259 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:00:35.809163 25283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a62143be7684975a3d179a39444a380. Latest consensus state: current_term: 1 leader_uuid: "0a62143be7684975a3d179a39444a380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.809276 25283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:00:35.818126 25298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a62143be7684975a3d179a39444a380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.818277 25298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:35.819505 25283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a62143be7684975a3d179a39444a380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.819610 25283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:00:35.820132 25301 mvcc.cc:205] Tried to move back new op lower bound from 7298467577014824960 to 7298467576972664832. Current Snapshot: MvccSnapshot[applied={T|T < 7298467577014824960}]
I20260619 07:00:35.821048 25259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a62143be7684975a3d179a39444a380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.821146 25259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380 [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:35.822028 25265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a62143be7684975a3d179a39444a380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "651df1cfe5674ae8aab647fa7ba3a5bd" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35445 } } peers { permanent_uuid: "0a62143be7684975a3d179a39444a380" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 45531 } } peers { permanent_uuid: "90f81baea69346558019201f49224c55" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36473 } } }
I20260619 07:00:35.822135 25265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:00:35.822824 25304 mvcc.cc:205] Tried to move back new op lower bound from 7298467577014824960 to 7298467576972664832. Current Snapshot: MvccSnapshot[applied={T|T < 7298467577014824960}]
I20260619 07:00:35.823710 25302 mvcc.cc:205] Tried to move back new op lower bound from 7298467577014824960 to 7298467576972664832. Current Snapshot: MvccSnapshot[applied={T|T < 7298467577014824960}]
I20260619 07:00:35.829155 25300 catalog_manager.cc:1347] Generated new cluster ID: 89aa6dc87ac84993bf0f6da8583d376c
I20260619 07:00:35.829221 25300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:00:35.858070 25300 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:00:35.859470 25300 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:00:35.890067 25300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a62143be7684975a3d179a39444a380: Generated new TSK 0
I20260619 07:00:35.890969 25300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:00:36.749343 25294 catalog_manager.cc:1259] Loaded cluster ID: 89aa6dc87ac84993bf0f6da8583d376c
I20260619 07:00:36.749434 25282 catalog_manager.cc:1259] Loaded cluster ID: 89aa6dc87ac84993bf0f6da8583d376c
I20260619 07:00:36.749454 25294 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd: loading cluster ID for follower catalog manager: success
I20260619 07:00:36.749518 25282 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55: loading cluster ID for follower catalog manager: success
I20260619 07:00:36.751112 25294 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd: acquiring CA information for follower catalog manager: success
I20260619 07:00:36.751153 25282 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55: acquiring CA information for follower catalog manager: success
I20260619 07:00:36.752208 25294 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 651df1cfe5674ae8aab647fa7ba3a5bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 07:00:36.752208 25282 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90f81baea69346558019201f49224c55: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 07:00:36.966658 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 25024
I20260619 07:00:36.973327 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 25099
I20260619 07:00:36.979593 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 25173
2026-06-19T07:00:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3122 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:00:37 dist-test-slave-10d0 krb5kdc[25315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:00:37 dist-test-slave-10d0 krb5kdc[25315](info): set up 2 sockets
Jun 19 07:00:37 dist-test-slave-10d0 krb5kdc[25315](info): commencing operation
krb5kdc: starting...
W20260619 07:00:38.021387 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:00:38 dist-test-slave-10d0 krb5kdc[25315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:00:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:00:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:00:38.144052 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:39947
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:32917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:39947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:38.255962 25328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:38.256229 25328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:38.256302 25328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:38.260340 25328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:38.260399 25328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:38.260419 25328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:38.260479 25328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:38.260505 25328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:38.265892 25328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:32917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:39947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:39947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:38.266943 25328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:38.268056 25328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:38.276439 25334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:38.276451 25336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:38.276612 25333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:38.276684 25328 server_base.cc:1034] running on GCE node
I20260619 07:00:38.277462 25328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:38.278579 25328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:38.279768 25328 hybrid_clock.cc:648] HybridClock initialized: now 1781852438279749 us; error 30 us; skew 500 ppm
Jun 19 07:00:38 dist-test-slave-10d0 krb5kdc[25315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852438, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:38.282388 25328 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:00:38.283268 25328 webserver.cc:460] Webserver started at http://127.11.106.126:46491/ using document root <none> and password file <none>
I20260619 07:00:38.283829 25328 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:38.283919 25328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:38.284157 25328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:38.286211 25328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "85feecdfc7a14f899b1c34adf62b40df"
format_stamp: "Formatted at 2026-06-19 07:00:38 on dist-test-slave-10d0"
server_key: "471f7b98d098496007d3e49ef073662d"
server_key_iv: "c39652a1b77e3db8274bf8300b87d07f"
server_key_version: "encryptionkey@0"
I20260619 07:00:38.286758 25328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "85feecdfc7a14f899b1c34adf62b40df"
format_stamp: "Formatted at 2026-06-19 07:00:38 on dist-test-slave-10d0"
server_key: "471f7b98d098496007d3e49ef073662d"
server_key_iv: "c39652a1b77e3db8274bf8300b87d07f"
server_key_version: "encryptionkey@0"
I20260619 07:00:38.290345 25328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 07:00:38.292654 25343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:38.293820 25328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 07:00:38.293960 25328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "85feecdfc7a14f899b1c34adf62b40df"
format_stamp: "Formatted at 2026-06-19 07:00:38 on dist-test-slave-10d0"
server_key: "471f7b98d098496007d3e49ef073662d"
server_key_iv: "c39652a1b77e3db8274bf8300b87d07f"
server_key_version: "encryptionkey@0"
I20260619 07:00:38.294075 25328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:38.305547 25328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:38.308266 25328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:38.308473 25328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:38.315960 25328 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:39947
I20260619 07:00:38.315989 25395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:39947 every 8 connection(s)
I20260619 07:00:38.316951 25328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:00:38.319895 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 25328
I20260619 07:00:38.319960 25396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:38.320044 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:00:38.320415 11689 external_mini_cluster.cc:1428] Setting key 6d3551b2fab2634a2df9ceb4da594c07
I20260619 07:00:38.326838 25396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df: Bootstrap starting.
Jun 19 07:00:38 dist-test-slave-10d0 krb5kdc[25315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:38.329632 25396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:38.330602 25396 log.cc:824] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:38.332942 25396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df: No bootstrap required, opened a new log
I20260619 07:00:38.335055 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:38.321855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.11.106.126:39947)
0619 07:00:38.322359 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:38.322375 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:38.322417 (+ 42us) server_negotiation.cc:407] Connection header received
0619 07:00:38.323066 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:38.323083 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:38.323401 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:38.323809 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:38.324658 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:38.325642 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:38.326548 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:38.327079 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:38.327253 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:38.330221 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:38.330226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:38.330240 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:38.330269 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:38.332059 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:38.332650 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:38.332656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:38.332661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:38.332776 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:38.333116 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:38.333121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:38.333122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:38.333515 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:38.333685 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:38.333995 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 07:00:38.334320 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":162,"threads_started":1}
I20260619 07:00:38.337087 25396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85feecdfc7a14f899b1c34adf62b40df" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39947 } }
I20260619 07:00:38.337402 25396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:38.337471 25396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85feecdfc7a14f899b1c34adf62b40df, State: Initialized, Role: FOLLOWER
I20260619 07:00:38.338114 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:00:38.338059 25396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85feecdfc7a14f899b1c34adf62b40df" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39947 } }
I20260619 07:00:38.338241 25396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:00:38.338310 25396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:00:38.338407 25396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:38.339668 25396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85feecdfc7a14f899b1c34adf62b40df" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39947 } }
I20260619 07:00:38.340050 25396 leader_election.cc:302] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85feecdfc7a14f899b1c34adf62b40df; no voters:
I20260619 07:00:38.340368 25396 leader_election.cc:288] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:00:38.340590 25401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:00:38.340881 25401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [term 1 LEADER]: Becoming Leader. State: Replica: 85feecdfc7a14f899b1c34adf62b40df, State: Running, Role: LEADER
I20260619 07:00:38.341207 25401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85feecdfc7a14f899b1c34adf62b40df" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39947 } }
I20260619 07:00:38.341719 25396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:38.342834 25402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85feecdfc7a14f899b1c34adf62b40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85feecdfc7a14f899b1c34adf62b40df" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39947 } } }
I20260619 07:00:38.342965 25402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:38.343276 25403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85feecdfc7a14f899b1c34adf62b40df. Latest consensus state: current_term: 1 leader_uuid: "85feecdfc7a14f899b1c34adf62b40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85feecdfc7a14f899b1c34adf62b40df" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39947 } } }
I20260619 07:00:38.343376 25410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:00:38.343433 25403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:38.346163 25410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:00:38.351754 25410 catalog_manager.cc:1347] Generated new cluster ID: 3863d36bcb604033b9a34d95572d4b64
I20260619 07:00:38.351822 25410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:00:38.357617 25410 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:00:38.358464 25410 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:00:38.364336 25410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85feecdfc7a14f899b1c34adf62b40df: Generated new TSK 0
I20260619 07:00:38.365053 25410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:00:38.394616 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 25328
2026-06-19T07:00:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1413 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:00:38 dist-test-slave-10d0 krb5kdc[25423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:00:38 dist-test-slave-10d0 krb5kdc[25423](info): set up 2 sockets
Jun 19 07:00:38 dist-test-slave-10d0 krb5kdc[25423](info): commencing operation
krb5kdc: starting...
W20260619 07:00:39.433338 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:00:39 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:00:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:00:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:00:39.560698 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42625
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:39.676684 25436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:39.676982 25436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:39.677028 25436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:39.681679 25436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:39.681761 25436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:39.681784 25436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:39.681804 25436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:39.681823 25436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:39.687237 25436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:39.688309 25436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:39.689421 25436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:39.698244 25444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:39.698251 25442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:39.698267 25441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:39.698485 25436 server_base.cc:1034] running on GCE node
I20260619 07:00:39.699226 25436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:39.700367 25436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:39.701625 25436 hybrid_clock.cc:648] HybridClock initialized: now 1781852439701611 us; error 40 us; skew 500 ppm
Jun 19 07:00:39 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852439, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:39.710598 25436 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:00:39.711591 25436 webserver.cc:460] Webserver started at http://127.11.106.126:43075/ using document root <none> and password file <none>
I20260619 07:00:39.712248 25436 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:39.712340 25436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:39.712651 25436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:39.714764 25436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "745ac748a8454c959c5940543d72902e"
format_stamp: "Formatted at 2026-06-19 07:00:39 on dist-test-slave-10d0"
server_key: "de6ce5f977fd1996f39a6857d63ccf8a"
server_key_iv: "a8a903abdcd33329bdfc4176d1ee8d67"
server_key_version: "encryptionkey@0"
I20260619 07:00:39.715337 25436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "745ac748a8454c959c5940543d72902e"
format_stamp: "Formatted at 2026-06-19 07:00:39 on dist-test-slave-10d0"
server_key: "de6ce5f977fd1996f39a6857d63ccf8a"
server_key_iv: "a8a903abdcd33329bdfc4176d1ee8d67"
server_key_version: "encryptionkey@0"
I20260619 07:00:39.719306 25436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 07:00:39.721834 25451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:39.723001 25436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:00:39.723176 25436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "745ac748a8454c959c5940543d72902e"
format_stamp: "Formatted at 2026-06-19 07:00:39 on dist-test-slave-10d0"
server_key: "de6ce5f977fd1996f39a6857d63ccf8a"
server_key_iv: "a8a903abdcd33329bdfc4176d1ee8d67"
server_key_version: "encryptionkey@0"
I20260619 07:00:39.723294 25436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:39.752211 25436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:39.755741 25436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:39.755955 25436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:39.762964 25436 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:42625
I20260619 07:00:39.763259 25503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:42625 every 8 connection(s)
I20260619 07:00:39.763963 25436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:00:39.766703 25504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:39.768039 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 25436
I20260619 07:00:39.768139 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:00:39.768440 11689 external_mini_cluster.cc:1428] Setting key f446cfd35dd733bcd9b0427dfc16e5a0
I20260619 07:00:39.773170 25504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Bootstrap starting.
I20260619 07:00:39.775750 25504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:39.776635 25504 log.cc:824] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Log is configured to *not* fsync() on all Append() calls
Jun 19 07:00:39 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:39.778709 25504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: No bootstrap required, opened a new log
I20260619 07:00:39.783394 25504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:39.783677 25504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:39.783733 25504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac748a8454c959c5940543d72902e, State: Initialized, Role: FOLLOWER
I20260619 07:00:39.784211 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:39.769985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.11.106.126:42625)
0619 07:00:39.770402 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:39.770424 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:39.770469 (+ 45us) server_negotiation.cc:407] Connection header received
0619 07:00:39.771329 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:39.771354 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:39.771784 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:39.772168 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:39.773077 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:39.774544 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:39.775403 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:39.776101 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:39.776268 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:39.779008 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:39.779016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:39.779036 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:39.779072 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:39.781050 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:39.781476 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:39.781482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:39.781489 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:39.781618 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:39.781937 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:39.781940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:39.781942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:39.782452 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:39.782655 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:39.782932 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 07:00:39.783237 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":96,"threads_started":1}
I20260619 07:00:39.784202 25504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:39.784371 25504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:00:39.784435 25504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:00:39.784555 25504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:39.785652 25504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:39.786005 25504 leader_election.cc:302] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 745ac748a8454c959c5940543d72902e; no voters:
I20260619 07:00:39.786317 25504 leader_election.cc:288] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:00:39.786424 25509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:00:39.786676 25509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 1 LEADER]: Becoming Leader. State: Replica: 745ac748a8454c959c5940543d72902e, State: Running, Role: LEADER
I20260619 07:00:39.786983 25509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:39.787554 25504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:39.788143 25509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745ac748a8454c959c5940543d72902e. Latest consensus state: current_term: 1 leader_uuid: "745ac748a8454c959c5940543d72902e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } } }
I20260619 07:00:39.788316 25509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:39.788614 25510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "745ac748a8454c959c5940543d72902e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } } }
I20260619 07:00:39.788714 25510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:39.788780 25516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:00:39.791575 25516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:00:39.797237 25516 catalog_manager.cc:1347] Generated new cluster ID: e3e37d674837441c93dda339226b7db2
I20260619 07:00:39.797317 25516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:00:39.818435 25516 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:00:39.820243 25516 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:00:39.829538 25516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Generated new TSK 0
I20260619 07:00:39.830232 25516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 07:00:39.866168 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42625
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:39.989326 25530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:39.989594 25530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:39.989697 25530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:39.994397 25530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:39.994463 25530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:39.994521 25530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:00:39.999758 25530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:40.000761 25530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:40.001758 25530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:40.011355 25535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:40.011365 25536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:40.011379 25538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:40.011838 25530 server_base.cc:1034] running on GCE node
I20260619 07:00:40.012256 25530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:40.012949 25530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:40.014103 25530 hybrid_clock.cc:648] HybridClock initialized: now 1781852440014068 us; error 48 us; skew 500 ppm
Jun 19 07:00:40 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:40.016856 25530 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:00:40.017695 25530 webserver.cc:460] Webserver started at http://127.11.106.65:39169/ using document root <none> and password file <none>
I20260619 07:00:40.018280 25530 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:40.018373 25530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:40.018599 25530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:40.020673 25530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "a105740decb74dd9b744333f46ae4a58"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6682879e2995578fdd03035f43532d73"
server_key_iv: "753cbbe3f23fcac02ceeec67674480bb"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.021245 25530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "a105740decb74dd9b744333f46ae4a58"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6682879e2995578fdd03035f43532d73"
server_key_iv: "753cbbe3f23fcac02ceeec67674480bb"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.024940 25530 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:00:40.027458 25545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.028895 25530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:00:40.028988 25530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "a105740decb74dd9b744333f46ae4a58"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6682879e2995578fdd03035f43532d73"
server_key_iv: "753cbbe3f23fcac02ceeec67674480bb"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.029073 25530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:40.041855 25530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:40.044742 25530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:40.044891 25530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:40.045404 25530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:40.046175 25530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:40.046222 25530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.046262 25530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:40.046283 25530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.055161 25530 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:45663
I20260619 07:00:40.055249 25658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:45663 every 8 connection(s)
I20260619 07:00:40.056118 25530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:00:40.064013 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 25530
I20260619 07:00:40.064167 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 07:00:40.064628 11689 external_mini_cluster.cc:1428] Setting key 4ca8adb403bf7da5f729297569790759
Jun 19 07:00:40 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:40.070220 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.057973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:60135 (local address 127.11.106.126:42625)
0619 07:00:40.058131 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.058136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.059048 (+ 912us) server_negotiation.cc:407] Connection header received
0619 07:00:40.060319 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.060322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.060368 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.060444 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:40.061616 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.062158 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.063245 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.063515 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.063610 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.066577 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:40.066580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:40.066583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:40.066609 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:40.068029 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:40.068636 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:40.068640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:40.068642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:40.068733 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:40.069134 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:40.069137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:40.069139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:40.069325 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:40.069434 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.069923 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.070047 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 07:00:40.070987 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.058422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:60135)
0619 07:00:40.058895 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:40.058936 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:40.060125 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:40.060670 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:40.060677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:40.061019 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:40.061479 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.061491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.062265 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.062268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.063111 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.063120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.063651 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.063654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.063803 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.064421 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:40.064442 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:40.066413 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:40.068195 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:40.068201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:40.068215 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:40.068471 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:40.068841 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:40.068844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:40.068846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:40.068990 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:40.069433 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:40.069439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:40.069682 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 07:00:40.069866 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 07:00:40.070190 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":120,"threads_started":1}
I20260619 07:00:40.072247 25659 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42625
I20260619 07:00:40.072506 25659 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:40.073110 25659 heartbeater.cc:511] Master 127.11.106.126:42625 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 07:00:40.074841 25468 ts_manager.cc:194] Registered new tserver with Master: a105740decb74dd9b744333f46ae4a58 (127.11.106.65:45663)
I20260619 07:00:40.075939 25468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:60135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 07:00:40.091885 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42625
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:40.201457 25664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:40.201742 25664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:40.201787 25664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:40.205796 25664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:40.205857 25664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:40.205917 25664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:00:40.211789 25664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:40.213008 25664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:40.214464 25664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:40.224843 25670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:40.224875 25672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:40.224872 25669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:40.225276 25664 server_base.cc:1034] running on GCE node
I20260619 07:00:40.225790 25664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:40.226577 25664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:40.227746 25664 hybrid_clock.cc:648] HybridClock initialized: now 1781852440227728 us; error 27 us; skew 500 ppm
Jun 19 07:00:40 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:40.230461 25664 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:00:40.231276 25664 webserver.cc:460] Webserver started at http://127.11.106.66:37481/ using document root <none> and password file <none>
I20260619 07:00:40.231827 25664 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:40.231884 25664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:40.232048 25664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:40.234143 25664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "ae0e048281184851bc745fda2e8b3d85"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "0772ae393e6279154644c47e523406a3"
server_key_iv: "48b28b33b3ccede80bdcad468e041e8f"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.234637 25664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "ae0e048281184851bc745fda2e8b3d85"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "0772ae393e6279154644c47e523406a3"
server_key_iv: "48b28b33b3ccede80bdcad468e041e8f"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.238382 25664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 07:00:40.240906 25679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.242307 25664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 07:00:40.242424 25664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "ae0e048281184851bc745fda2e8b3d85"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "0772ae393e6279154644c47e523406a3"
server_key_iv: "48b28b33b3ccede80bdcad468e041e8f"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.242524 25664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:40.254473 25664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:40.257782 25664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:40.258010 25664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:40.258675 25664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:40.259541 25664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:40.259621 25664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.259693 25664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:40.259747 25664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.269023 25664 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:46083
I20260619 07:00:40.269066 25792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:46083 every 8 connection(s)
I20260619 07:00:40.269976 25664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:00:40.278134 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 25664
I20260619 07:00:40.278316 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 07:00:40.278606 11689 external_mini_cluster.cc:1428] Setting key 2d5884131448533f6c6eee54781e2c89
Jun 19 07:00:40 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:40.284391 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.271865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:39909 (local address 127.11.106.126:42625)
0619 07:00:40.272045 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.272050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.273002 (+ 952us) server_negotiation.cc:407] Connection header received
0619 07:00:40.274335 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.274338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.274385 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.274465 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:40.275656 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.276247 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.277413 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.277669 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.277762 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.280733 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:40.280736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:40.280738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:40.280760 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:40.282336 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:40.282839 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:40.282843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:40.282845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:40.282925 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:40.283268 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:40.283272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:40.283273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:40.283452 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:40.283558 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.284111 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.284233 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 07:00:40.285281 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.272301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:39909)
0619 07:00:40.272840 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:40.272883 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:40.274105 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:40.274652 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:40.274659 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:40.275008 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:40.275486 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.275499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.276362 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.276365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.277286 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.277295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.277882 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.277888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.278096 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.278789 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:40.278810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:40.280540 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:40.282457 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:40.282463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:40.282486 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:40.282731 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:40.283030 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:40.283034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:40.283035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:40.283167 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:40.283580 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:40.283587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:40.283843 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 07:00:40.284057 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 07:00:40.284387 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":185,"threads_started":1}
I20260619 07:00:40.286510 25793 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42625
I20260619 07:00:40.286760 25793 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:40.287237 25793 heartbeater.cc:511] Master 127.11.106.126:42625 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 07:00:40.288341 25468 ts_manager.cc:194] Registered new tserver with Master: ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083)
I20260619 07:00:40.289027 25468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:39909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 07:00:40.307303 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42625
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:40.418515 25798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:40.418793 25798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:40.418838 25798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:40.423041 25798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:40.423099 25798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:40.423157 25798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:00:40.428395 25798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:40.429433 25798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:40.430377 25798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:40.440342 25804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:40.440418 25806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:40.440346 25803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:40.440410 25798 server_base.cc:1034] running on GCE node
I20260619 07:00:40.440984 25798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:40.441632 25798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:40.442797 25798 hybrid_clock.cc:648] HybridClock initialized: now 1781852440442759 us; error 46 us; skew 500 ppm
Jun 19 07:00:40 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:40.446239 25798 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:00:40.447149 25798 webserver.cc:460] Webserver started at http://127.11.106.67:42729/ using document root <none> and password file <none>
I20260619 07:00:40.447669 25798 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:40.447727 25798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:40.447890 25798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:40.449937 25798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "adc686a0f205482baa6f8fe4f373cf7b"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6d6f8bcb457ea997286bceb9d691c87b"
server_key_iv: "5486c0a64e2bc305b91e430201754400"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.450454 25798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "adc686a0f205482baa6f8fe4f373cf7b"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6d6f8bcb457ea997286bceb9d691c87b"
server_key_iv: "5486c0a64e2bc305b91e430201754400"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.454146 25798 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 07:00:40.456583 25813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.457803 25798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:00:40.457944 25798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "adc686a0f205482baa6f8fe4f373cf7b"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6d6f8bcb457ea997286bceb9d691c87b"
server_key_iv: "5486c0a64e2bc305b91e430201754400"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.458078 25798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:40.508288 25798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:40.511441 25798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:40.511704 25798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:40.512336 25798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:40.513248 25798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:40.513320 25798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.513392 25798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:40.513445 25798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.522856 25798 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:43069
I20260619 07:00:40.522891 25926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:43069 every 8 connection(s)
I20260619 07:00:40.523777 25798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:00:40.524158 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 25798
I20260619 07:00:40.524304 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 07:00:40.524663 11689 external_mini_cluster.cc:1428] Setting key 4745a1e16f5483bd0241e493fcbbe251
Jun 19 07:00:40 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:40.537746 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.526020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:39801 (local address 127.11.106.126:42625)
0619 07:00:40.526183 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.526186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.527218 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 07:00:40.528590 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.528593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.528645 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.528716 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:40.529881 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.530439 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.531571 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.531807 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.531921 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.534527 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:40.534530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:40.534532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:40.534557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:40.535782 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:40.536226 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:40.536229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:40.536230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:40.536300 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:40.536648 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:40.536650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:40.536652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:40.536816 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:40.536895 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.537465 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.537584 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 07:00:40.538786 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.526513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:39801)
0619 07:00:40.527080 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:40.527124 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:40.528358 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:40.528887 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:40.528895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:40.529249 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:40.529735 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.529748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.530571 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.530574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.531444 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.531453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.531960 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.531964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.532135 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.532909 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:40.532926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:40.534377 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:40.535886 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:40.535890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:40.535902 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:40.536142 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:40.536376 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:40.536378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:40.536379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:40.536510 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:40.536939 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:40.536944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:40.537174 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 07:00:40.537403 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 07:00:40.537820 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":157,"threads_started":1}
I20260619 07:00:40.539868 25927 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42625
I20260619 07:00:40.540099 25927 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:40.540654 25927 heartbeater.cc:511] Master 127.11.106.126:42625 requested a full tablet report, sending...
I20260619 07:00:40.541657 25468 ts_manager.cc:194] Registered new tserver with Master: adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069)
I20260619 07:00:40.542312 25468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:39801
I20260619 07:00:40.550549 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:00:40.559705 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.552500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.11.106.126:42625)
0619 07:00:40.552707 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.552711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.552802 (+ 91us) server_negotiation.cc:407] Connection header received
0619 07:00:40.552906 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.552908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.552946 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.553020 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:40.553724 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.554248 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.555061 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.555276 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.555368 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.556683 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:40.556686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:40.556688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:40.556712 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:40.558057 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:40.558516 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:40.558519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:40.558521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:40.558615 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:40.558875 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:40.558879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:40.558880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:40.559113 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:40.559234 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.559436 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.559574 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 07:00:40.562763 25468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:00:40.565783 25468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:00:40.580005 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.574996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33182 (local address 127.11.106.66:46083)
0619 07:00:40.575309 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.575321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.575476 (+ 155us) server_negotiation.cc:407] Connection header received
0619 07:00:40.575611 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.575617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.575848 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.576024 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:40.576317 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.577149 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.578428 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.579251 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.579366 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.579539 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.579767 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.579857 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":132,"threads_started":1}
I20260619 07:00:40.580005 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.574539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:46083 (local address 127.0.0.1:33182)
0619 07:00:40.575381 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:40.575392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:40.575496 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:40.576018 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:40.576024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:40.576047 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:40.576185 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.576192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.577289 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.577295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.578292 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.578301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.579453 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.579457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.579563 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.579607 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 07:00:40.579736 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 07:00:40.579852 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":86,"threads_started":1}
I20260619 07:00:40.580899 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.574200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43069 (local address 127.0.0.1:50046)
0619 07:00:40.574563 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:40.574625 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:40.574746 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:40.576282 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:40.576286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:40.576305 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:40.576396 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.576401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.578117 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.578123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.579183 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.579193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.580272 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.580276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.580623 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.580642 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 07:00:40.580712 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 07:00:40.580774 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":205,"thread_start_us":81,"threads_started":1}
I20260619 07:00:40.580935 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.575021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.11.106.67:43069)
0619 07:00:40.575799 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.575809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.575829 (+ 20us) server_negotiation.cc:407] Connection header received
0619 07:00:40.575893 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.575897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.576067 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.576249 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:40.576832 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.577960 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.579303 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.580122 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.580410 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.580524 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.580748 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.580823 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":151,"threads_started":1}
I20260619 07:00:40.581250 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.574216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45663 (local address 127.0.0.1:40262)
0619 07:00:40.575006 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:40.575020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:40.575107 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:40.576520 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:40.576524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:40.576561 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:40.576680 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.576687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.578095 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.578098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.579306 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.579320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.580272 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.580276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.580460 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.580487 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 07:00:40.581063 (+ 576us) client_negotiation.cc:240] Negotiation successful
0619 07:00:40.581128 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":83,"thread_start_us":95,"threads_started":1}
I20260619 07:00:40.581723 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.575395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.11.106.65:45663)
0619 07:00:40.575650 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.575656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.575681 (+ 25us) server_negotiation.cc:407] Connection header received
0619 07:00:40.575757 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.575762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.576371 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.576538 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:40.576795 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.577960 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.579422 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.580146 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.580366 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.580506 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.581498 (+ 992us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.581593 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260619 07:00:40.583390 25727 tablet_service.cc:1467] Processing CreateTablet for tablet 2df19a7bb3c44317816d0a2b7f9c1db1 (DEFAULT_TABLE table=test-table [id=2c8e954a716b4fa6a85be5631edfe600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:40.583573 25861 tablet_service.cc:1467] Processing CreateTablet for tablet 2df19a7bb3c44317816d0a2b7f9c1db1 (DEFAULT_TABLE table=test-table [id=2c8e954a716b4fa6a85be5631edfe600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:40.584477 25861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df19a7bb3c44317816d0a2b7f9c1db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:40.584510 25727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df19a7bb3c44317816d0a2b7f9c1db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:40.584314 25593 tablet_service.cc:1467] Processing CreateTablet for tablet 2df19a7bb3c44317816d0a2b7f9c1db1 (DEFAULT_TABLE table=test-table [id=2c8e954a716b4fa6a85be5631edfe600]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:40.585261 25593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df19a7bb3c44317816d0a2b7f9c1db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:40.591617 25944 tablet_bootstrap.cc:492] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: Bootstrap starting.
I20260619 07:00:40.592875 25946 tablet_bootstrap.cc:492] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Bootstrap starting.
I20260619 07:00:40.592890 25945 tablet_bootstrap.cc:492] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: Bootstrap starting.
I20260619 07:00:40.594458 25944 tablet_bootstrap.cc:654] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:40.594731 25946 tablet_bootstrap.cc:654] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:40.595413 25945 tablet_bootstrap.cc:654] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:40.595424 25944 log.cc:824] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:40.595435 25946 log.cc:824] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:40.596222 25945 log.cc:824] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:40.597831 25944 tablet_bootstrap.cc:492] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: No bootstrap required, opened a new log
I20260619 07:00:40.598062 25946 tablet_bootstrap.cc:492] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: No bootstrap required, opened a new log
I20260619 07:00:40.598054 25944 ts_tablet_manager.cc:1397] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260619 07:00:40.598259 25945 tablet_bootstrap.cc:492] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: No bootstrap required, opened a new log
I20260619 07:00:40.598284 25946 ts_tablet_manager.cc:1397] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 07:00:40.598446 25945 ts_tablet_manager.cc:1397] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 07:00:40.601675 25946 raft_consensus.cc:348] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:40.601926 25946 raft_consensus.cc:374] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:40.601998 25946 raft_consensus.cc:729] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a105740decb74dd9b744333f46ae4a58, State: Initialized, Role: FOLLOWER
I20260619 07:00:40.602216 25945 raft_consensus.cc:348] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:40.602186 25944 raft_consensus.cc:348] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:40.602461 25945 raft_consensus.cc:374] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:40.602473 25944 raft_consensus.cc:374] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:40.602537 25944 raft_consensus.cc:729] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc686a0f205482baa6f8fe4f373cf7b, State: Initialized, Role: FOLLOWER
I20260619 07:00:40.602483 25946 consensus_queue.cc:260] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:40.602576 25945 raft_consensus.cc:729] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0e048281184851bc745fda2e8b3d85, State: Initialized, Role: FOLLOWER
I20260619 07:00:40.603004 25945 consensus_queue.cc:260] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:40.603004 25944 consensus_queue.cc:260] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:40.603518 25946 ts_tablet_manager.cc:1428] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 07:00:40.603952 25659 heartbeater.cc:503] Master 127.11.106.126:42625 was elected leader, sending a full tablet report...
I20260619 07:00:40.603997 25945 ts_tablet_manager.cc:1428] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 07:00:40.604308 25793 heartbeater.cc:503] Master 127.11.106.126:42625 was elected leader, sending a full tablet report...
I20260619 07:00:40.604331 25927 heartbeater.cc:503] Master 127.11.106.126:42625 was elected leader, sending a full tablet report...
I20260619 07:00:40.605469 25944 ts_tablet_manager.cc:1428] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 07:00:40.720906 25951 raft_consensus.cc:482] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:40.721251 25951 raft_consensus.cc:504] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:40.722527 25951 leader_election.cc:288] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069), a105740decb74dd9b744333f46ae4a58 (127.11.106.65:45663)
I20260619 07:00:40.726526 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.722856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:59657 (local address 127.11.106.67:43069)
0619 07:00:40.723211 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.723216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.723234 (+ 18us) server_negotiation.cc:407] Connection header received
0619 07:00:40.723288 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.723290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.723327 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.723414 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:40.723694 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.724196 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.725578 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.725973 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.726115 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.726171 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.726351 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.726418 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260619 07:00:40.726542 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.722791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43069 (local address 127.11.106.66:59657)
0619 07:00:40.722997 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:40.723014 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:40.723114 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:40.723428 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:40.723431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:40.723458 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:40.723583 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.723588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.724312 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.724317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.725463 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.725474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.726102 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.726105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.726210 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.726232 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 07:00:40.726326 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 07:00:40.726395 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":24,"spinlock_wait_cycles":4864}
I20260619 07:00:40.726902 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.723014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:33517 (local address 127.11.106.65:45663)
0619 07:00:40.723188 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.723193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.723437 (+ 244us) server_negotiation.cc:407] Connection header received
0619 07:00:40.723606 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.723611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.723658 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.723745 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:40.724022 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.724508 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.725834 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.726311 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.726440 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.726511 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.726710 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.726778 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 07:00:40.727205 25881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df19a7bb3c44317816d0a2b7f9c1db1" candidate_uuid: "ae0e048281184851bc745fda2e8b3d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc686a0f205482baa6f8fe4f373cf7b" is_pre_election: true
I20260619 07:00:40.727295 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.722871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45663 (local address 127.11.106.66:33517)
0619 07:00:40.723326 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:40.723336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:40.723478 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:40.723757 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:40.723760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:40.723776 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:40.723906 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.723911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.724661 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.724668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.725715 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:40.725724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.726431 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.726435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:40.726529 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.726568 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 07:00:40.727134 (+ 566us) client_negotiation.cc:240] Negotiation successful
0619 07:00:40.727192 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":1,"spinlock_wait_cycles":4736,"thread_start_us":163,"threads_started":1}
I20260619 07:00:40.727558 25881 raft_consensus.cc:2454] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0e048281184851bc745fda2e8b3d85 in term 0.
I20260619 07:00:40.727948 25613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df19a7bb3c44317816d0a2b7f9c1db1" candidate_uuid: "ae0e048281184851bc745fda2e8b3d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a105740decb74dd9b744333f46ae4a58" is_pre_election: true
I20260619 07:00:40.728032 25680 leader_election.cc:302] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc686a0f205482baa6f8fe4f373cf7b, ae0e048281184851bc745fda2e8b3d85; no voters:
I20260619 07:00:40.728235 25613 raft_consensus.cc:2454] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0e048281184851bc745fda2e8b3d85 in term 0.
I20260619 07:00:40.728333 25951 raft_consensus.cc:2781] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:00:40.728422 25951 raft_consensus.cc:482] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:00:40.728519 25951 raft_consensus.cc:3037] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:40.729560 25951 raft_consensus.cc:504] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:40.729933 25951 leader_election.cc:288] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [CANDIDATE]: Term 1 election: Requested vote from peers adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069), a105740decb74dd9b744333f46ae4a58 (127.11.106.65:45663)
I20260619 07:00:40.730257 25881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df19a7bb3c44317816d0a2b7f9c1db1" candidate_uuid: "ae0e048281184851bc745fda2e8b3d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc686a0f205482baa6f8fe4f373cf7b"
I20260619 07:00:40.730342 25613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df19a7bb3c44317816d0a2b7f9c1db1" candidate_uuid: "ae0e048281184851bc745fda2e8b3d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a105740decb74dd9b744333f46ae4a58"
I20260619 07:00:40.730393 25881 raft_consensus.cc:3037] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:40.730507 25613 raft_consensus.cc:3037] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:40.731400 25881 raft_consensus.cc:2454] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0e048281184851bc745fda2e8b3d85 in term 1.
I20260619 07:00:40.731567 25613 raft_consensus.cc:2454] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0e048281184851bc745fda2e8b3d85 in term 1.
I20260619 07:00:40.731683 25680 leader_election.cc:302] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc686a0f205482baa6f8fe4f373cf7b, ae0e048281184851bc745fda2e8b3d85; no voters:
I20260619 07:00:40.731850 25951 raft_consensus.cc:2781] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:00:40.732105 25951 raft_consensus.cc:686] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 1 LEADER]: Becoming Leader. State: Replica: ae0e048281184851bc745fda2e8b3d85, State: Running, Role: LEADER
I20260619 07:00:40.732446 25951 consensus_queue.cc:237] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:40.735930 25468 catalog_manager.cc:5495] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0e048281184851bc745fda2e8b3d85 (127.11.106.66). New cstate: current_term: 1 leader_uuid: "ae0e048281184851bc745fda2e8b3d85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } health_report { overall_health: HEALTHY } } }
I20260619 07:00:40.771071 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.767104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.11.106.66:46083)
0619 07:00:40.767243 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:40.767248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:40.767263 (+ 15us) server_negotiation.cc:407] Connection header received
0619 07:00:40.767409 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:40.767412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:40.767504 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:40.767623 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:00:40.767888 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.768709 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.769511 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:40.769858 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:40.769951 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:40.770227 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:00:40.770714 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:00:40.770820 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:40.770868 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 07:00:40.770942 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 07:00:40.771118 25794 tablet.cc:2369] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 07:00:40.774893 25928 tablet.cc:2369] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:00:40.777128 25881 raft_consensus.cc:1261] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [term 1 FOLLOWER]: Refusing update from remote peer ae0e048281184851bc745fda2e8b3d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:00:40.777190 25613 raft_consensus.cc:1261] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Refusing update from remote peer ae0e048281184851bc745fda2e8b3d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:00:40.777841 25954 consensus_queue.cc:1035] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:00:40.778048 25951 consensus_queue.cc:1035] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:00:40.787710 25958 mvcc.cc:205] Tried to move back new op lower bound from 7298467597416251392 to 7298467597243219968. Current Snapshot: MvccSnapshot[applied={T|T < 7298467597416251392 or (T in {7298467597416251392})}]
I20260619 07:00:40.795477 25468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:00:40.795624 25468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:00:40.798077 25468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Sending DeleteTablet for 3 replicas of tablet 2df19a7bb3c44317816d0a2b7f9c1db1
I20260619 07:00:40.798857 25861 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df19a7bb3c44317816d0a2b7f9c1db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:40 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50046
I20260619 07:00:40.798923 25593 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df19a7bb3c44317816d0a2b7f9c1db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:40 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:40262
I20260619 07:00:40.799010 25727 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df19a7bb3c44317816d0a2b7f9c1db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:40 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:33182
I20260619 07:00:40.799443 25967 tablet_replica.cc:330] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: stopping tablet replica
I20260619 07:00:40.799549 25968 tablet_replica.cc:330] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: stopping tablet replica
I20260619 07:00:40.799576 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 25530
I20260619 07:00:40.799754 25969 tablet_replica.cc:330] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: stopping tablet replica
I20260619 07:00:40.799831 25967 raft_consensus.cc:2229] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:00:40.800004 25969 raft_consensus.cc:2229] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:00:40.800261 25967 raft_consensus.cc:2258] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:00:40.800359 25968 raft_consensus.cc:2229] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 1 LEADER]: Raft consensus shutting down.
I20260619 07:00:40.801210 25968 raft_consensus.cc:2258] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:00:40.801833 25967 ts_tablet_manager.cc:1905] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:00:40.803519 25968 ts_tablet_manager.cc:1905] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:00:40.805953 25968 ts_tablet_manager.cc:1918] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:00:40.806071 25968 log.cc:1196] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/2df19a7bb3c44317816d0a2b7f9c1db1
I20260619 07:00:40.806411 25968 ts_tablet_manager.cc:1939] T 2df19a7bb3c44317816d0a2b7f9c1db1 P ae0e048281184851bc745fda2e8b3d85: Deleting consensus metadata
I20260619 07:00:40.806927 25967 ts_tablet_manager.cc:1918] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:00:40.807090 25967 log.cc:1196] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/2df19a7bb3c44317816d0a2b7f9c1db1
I20260619 07:00:40.807464 25967 ts_tablet_manager.cc:1939] T 2df19a7bb3c44317816d0a2b7f9c1db1 P adc686a0f205482baa6f8fe4f373cf7b: Deleting consensus metadata
I20260619 07:00:40.807760 25454 catalog_manager.cc:4841] TS ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083): tablet 2df19a7bb3c44317816d0a2b7f9c1db1 (table test-table [id=2c8e954a716b4fa6a85be5631edfe600]) successfully deleted
W20260619 07:00:40.808869 25455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:45663 (error 108)
I20260619 07:00:40.809072 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 25664
I20260619 07:00:40.809644 25452 catalog_manager.cc:4841] TS adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069): tablet 2df19a7bb3c44317816d0a2b7f9c1db1 (table test-table [id=2c8e954a716b4fa6a85be5631edfe600]) successfully deleted
I20260619 07:00:40.810505 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:40.810133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45663 (local address 127.0.0.1:40264)
0619 07:00:40.810320 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:40.810399 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 07:00:40.810720 25455 catalog_manager.cc:4584] TS a105740decb74dd9b744333f46ae4a58 (127.11.106.65:45663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2df19a7bb3c44317816d0a2b7f9c1db1: Network error: Client connection negotiation failed: client connection to 127.11.106.65:45663: connect: Connection refused (error 111)
I20260619 07:00:40.815922 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 25798
I20260619 07:00:40.821772 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 25436
I20260619 07:00:40.828197 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42625
--webserver_interface=127.11.106.126
--webserver_port=43075
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:40.939419 25971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:40.939689 25971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:40.939729 25971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:40.943861 25971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:40.943918 25971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:40.943938 25971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:40.943977 25971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:40.944007 25971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:40.949388 25971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=43075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:40.950445 25971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:40.951586 25971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:40.966840 25980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:40.966840 25977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:40.966930 25971 server_base.cc:1034] running on GCE node
W20260619 07:00:40.966840 25978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:40.967669 25971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:40.968915 25971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:40.970094 25971 hybrid_clock.cc:648] HybridClock initialized: now 1781852440970055 us; error 52 us; skew 500 ppm
Jun 19 07:00:40 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:40.972784 25971 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:00:40.973692 25971 webserver.cc:460] Webserver started at http://127.11.106.126:43075/ using document root <none> and password file <none>
I20260619 07:00:40.974283 25971 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:40.974391 25971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:40.978267 25971 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 07:00:40.980388 25987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:40.981662 25971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 07:00:40.981803 25971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "745ac748a8454c959c5940543d72902e"
format_stamp: "Formatted at 2026-06-19 07:00:39 on dist-test-slave-10d0"
server_key: "de6ce5f977fd1996f39a6857d63ccf8a"
server_key_iv: "a8a903abdcd33329bdfc4176d1ee8d67"
server_key_version: "encryptionkey@0"
I20260619 07:00:40.982410 25971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:40.999167 25971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:41.002136 25971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:41.002357 25971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:41.009713 25971 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:42625
I20260619 07:00:41.010018 26039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:42625 every 8 connection(s)
I20260619 07:00:41.010901 25971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:00:41.014803 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 25971
I20260619 07:00:41.015236 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:45663
--local_ip_for_outbound_sockets=127.11.106.65
--tserver_master_addrs=127.11.106.126:42625
--webserver_port=39169
--webserver_interface=127.11.106.65
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:00:41.015802 26040 sys_catalog.cc:263] Verifying existing consensus state
I20260619 07:00:41.017571 26040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Bootstrap starting.
I20260619 07:00:41.034670 26040 log.cc:824] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:41.043767 26040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 07:00:41.044133 26040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Bootstrap complete.
I20260619 07:00:41.048229 26040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:41.048719 26040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac748a8454c959c5940543d72902e, State: Initialized, Role: FOLLOWER
I20260619 07:00:41.049199 26040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:41.049347 26040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:00:41.049439 26040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:00:41.049567 26040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 1 FOLLOWER]: Advancing to term 2
I20260619 07:00:41.051586 26040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:41.051934 26040 leader_election.cc:302] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 745ac748a8454c959c5940543d72902e; no voters:
I20260619 07:00:41.052263 26040 leader_election.cc:288] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 07:00:41.052418 26044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 07:00:41.052695 26044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 2 LEADER]: Becoming Leader. State: Replica: 745ac748a8454c959c5940543d72902e, State: Running, Role: LEADER
I20260619 07:00:41.053068 26044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:41.053508 26040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:41.054088 26044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745ac748a8454c959c5940543d72902e. Latest consensus state: current_term: 2 leader_uuid: "745ac748a8454c959c5940543d72902e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } } }
I20260619 07:00:41.054201 26044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:41.054551 26049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:00:41.054611 26045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "745ac748a8454c959c5940543d72902e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } } }
I20260619 07:00:41.054800 26045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:41.059255 26049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:00:41.060096 26049 catalog_manager.cc:1259] Loaded cluster ID: e3e37d674837441c93dda339226b7db2
I20260619 07:00:41.060168 26049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:00:41.061738 26049 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 07:00:41.066179 26049 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 07:00:41.071300 26049 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:00:41.072036 26049 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Loaded TSK: 0
I20260619 07:00:41.073289 26049 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 07:00:41.132226 26042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:41.132474 26042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:41.132517 26042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:41.136499 26042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:41.136615 26042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:41.136710 26042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:00:41.142043 26042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:45663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=39169
--tserver_master_addrs=127.11.106.126:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:41.143059 26042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:41.144197 26042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:41.155180 26068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:41.155212 26067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:41.155562 26070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:41.155733 26042 server_base.cc:1034] running on GCE node
I20260619 07:00:41.156229 26042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:41.157366 26042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:41.158612 26042 hybrid_clock.cc:648] HybridClock initialized: now 1781852441158577 us; error 50 us; skew 500 ppm
Jun 19 07:00:41 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:41.162046 26042 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:00:41.163180 26042 webserver.cc:460] Webserver started at http://127.11.106.65:39169/ using document root <none> and password file <none>
I20260619 07:00:41.163803 26042 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:41.163872 26042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:41.167973 26042 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 07:00:41.170343 26077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:41.171696 26042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 07:00:41.171829 26042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "a105740decb74dd9b744333f46ae4a58"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6682879e2995578fdd03035f43532d73"
server_key_iv: "753cbbe3f23fcac02ceeec67674480bb"
server_key_version: "encryptionkey@0"
I20260619 07:00:41.172382 26042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:41.197664 26042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:41.201159 26042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:41.201367 26042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:41.202054 26042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:41.203164 26084 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 07:00:41.207446 26042 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 07:00:41.207531 26042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 07:00:41.207614 26042 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 07:00:41.209867 26042 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 07:00:41.209940 26042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.001s
I20260619 07:00:41.210256 26084 tablet_bootstrap.cc:492] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Bootstrap starting.
I20260619 07:00:41.222368 26042 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:45663
I20260619 07:00:41.223009 26191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:45663 every 8 connection(s)
I20260619 07:00:41.223976 26042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:00:41.232350 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 26042
I20260619 07:00:41.232893 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:46083
--local_ip_for_outbound_sockets=127.11.106.66
--tserver_master_addrs=127.11.106.126:42625
--webserver_port=37481
--webserver_interface=127.11.106.66
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:00:41.234632 26084 log.cc:824] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Log is configured to *not* fsync() on all Append() calls
Jun 19 07:00:41 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:41.244316 26084 tablet_bootstrap.cc:492] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 07:00:41.244833 26084 tablet_bootstrap.cc:492] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Bootstrap complete.
I20260619 07:00:41.246385 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.226877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:45475 (local address 127.11.106.126:42625)
0619 07:00:41.229459 (+ 2582us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.229479 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.229523 (+ 44us) server_negotiation.cc:407] Connection header received
0619 07:00:41.229815 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.229830 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.230410 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.230805 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:41.231883 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.233480 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.235444 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.236211 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.236403 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.239718 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:41.239725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:41.239742 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:41.239777 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:41.242100 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.242808 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.242814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.242819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.242977 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.243444 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.243450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.243453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.243997 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:41.244124 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.245033 (+ 909us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.245373 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2421,"thread_start_us":149,"threads_started":1}
I20260619 07:00:41.247188 26084 ts_tablet_manager.cc:1397] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Time spent bootstrapping tablet: real 0.037s user 0.026s sys 0.008s
I20260619 07:00:41.248234 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.226889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:45475)
0619 07:00:41.227592 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.227650 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:41.228901 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:41.230853 (+ 1952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:41.230860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:41.231223 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:41.231697 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.231712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.233660 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.233666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.235299 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.235312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.236376 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.236380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.236591 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.237298 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:41.237327 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:41.239481 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:41.242249 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:41.242255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:41.242271 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:41.242668 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:41.243089 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:41.243095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:41.243100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:41.243302 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:41.244239 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:41.244247 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:41.244614 (+ 367us) client_negotiation.cc:769] Sending connection context
0619 07:00:41.246704 (+ 2090us) client_negotiation.cc:240] Negotiation successful
0619 07:00:41.247116 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":182,"threads_started":1}
I20260619 07:00:41.250900 26192 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42625
I20260619 07:00:41.251246 26192 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:41.251435 26084 raft_consensus.cc:348] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:41.251955 26084 raft_consensus.cc:729] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a105740decb74dd9b744333f46ae4a58, State: Initialized, Role: FOLLOWER
I20260619 07:00:41.252183 26192 heartbeater.cc:511] Master 127.11.106.126:42625 requested a full tablet report, sending...
I20260619 07:00:41.252475 26084 consensus_queue.cc:260] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } }
I20260619 07:00:41.253744 26084 ts_tablet_manager.cc:1428] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 07:00:41.256664 26004 ts_manager.cc:194] Registered new tserver with Master: a105740decb74dd9b744333f46ae4a58 (127.11.106.65:45663)
I20260619 07:00:41.260911 26004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:45475
I20260619 07:00:41.263535 26192 heartbeater.cc:503] Master 127.11.106.126:42625 was elected leader, sending a full tablet report...
Jun 19 07:00:41 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852440, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 07:00:41.271750 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.260272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.11.106.65:45663)
0619 07:00:41.261274 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.261283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.261303 (+ 20us) server_negotiation.cc:407] Connection header received
0619 07:00:41.261362 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.261366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.261519 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.261670 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:41.262807 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.263904 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.264986 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.265501 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.265629 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.268244 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:41.268254 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:41.268264 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:41.268286 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:41.269798 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.270295 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.270306 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.270307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.270386 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.270682 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.270686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.270688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.270930 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:41.271082 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.271428 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.271593 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":113,"threads_started":1}
I20260619 07:00:41.271750 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.260211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45663 (local address 127.0.0.1:40274)
0619 07:00:41.260840 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.260887 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:41.261030 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:41.262059 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:41.262066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:41.262510 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:41.262653 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.262661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.264038 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.264041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.264861 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.264871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.265636 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.265640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.265750 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.266345 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:41.266359 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:41.268081 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:41.269915 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:41.269926 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:41.269931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:41.270173 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:41.270491 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:41.270495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:41.270497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:41.270589 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:41.271048 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:41.271054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:41.271201 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 07:00:41.271347 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 07:00:41.271548 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":92,"threads_started":1}
I20260619 07:00:41.273092 26120 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df19a7bb3c44317816d0a2b7f9c1db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:40 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:40274
I20260619 07:00:41.273916 26201 tablet_replica.cc:330] stopping tablet replica
I20260619 07:00:41.274224 26201 raft_consensus.cc:2229] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:00:41.274483 26201 raft_consensus.cc:2258] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:00:41.275538 26201 ts_tablet_manager.cc:1905] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:00:41.279634 26201 ts_tablet_manager.cc:1918] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:00:41.279754 26201 log.cc:1196] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/2df19a7bb3c44317816d0a2b7f9c1db1
I20260619 07:00:41.280026 26201 ts_tablet_manager.cc:1939] T 2df19a7bb3c44317816d0a2b7f9c1db1 P a105740decb74dd9b744333f46ae4a58: Deleting consensus metadata
I20260619 07:00:41.281785 25991 catalog_manager.cc:4841] TS a105740decb74dd9b744333f46ae4a58 (127.11.106.65:45663): tablet 2df19a7bb3c44317816d0a2b7f9c1db1 (table test-table [id=2c8e954a716b4fa6a85be5631edfe600]) successfully deleted
W20260619 07:00:41.362133 26196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:41.362413 26196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:41.362502 26196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:41.366657 26196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:41.366742 26196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:41.366830 26196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:00:41.372047 26196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:46083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=37481
--tserver_master_addrs=127.11.106.126:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:41.373158 26196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:41.374215 26196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:41.383932 26206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:41.384043 26207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:41.384213 26196 server_base.cc:1034] running on GCE node
W20260619 07:00:41.384286 26209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:41.384719 26196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:41.385365 26196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:41.386615 26196 hybrid_clock.cc:648] HybridClock initialized: now 1781852441386536 us; error 70 us; skew 500 ppm
Jun 19 07:00:41 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:41.389358 26196 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:00:41.390255 26196 webserver.cc:460] Webserver started at http://127.11.106.66:37481/ using document root <none> and password file <none>
I20260619 07:00:41.390832 26196 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:41.390935 26196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:41.394757 26196 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 07:00:41.396874 26216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:41.398058 26196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 07:00:41.398200 26196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "ae0e048281184851bc745fda2e8b3d85"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "0772ae393e6279154644c47e523406a3"
server_key_iv: "48b28b33b3ccede80bdcad468e041e8f"
server_key_version: "encryptionkey@0"
I20260619 07:00:41.398770 26196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:41.421010 26196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:41.424026 26196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:41.424250 26196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:41.424896 26196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:41.425748 26196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:41.425822 26196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:41.425895 26196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:41.425930 26196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:41.435356 26196 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:46083
I20260619 07:00:41.435376 26329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:46083 every 8 connection(s)
I20260619 07:00:41.436410 26196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:00:41.440522 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 26196
I20260619 07:00:41.440999 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:43069
--local_ip_for_outbound_sockets=127.11.106.67
--tserver_master_addrs=127.11.106.126:42625
--webserver_port=42729
--webserver_interface=127.11.106.67
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 07:00:41 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:41.452929 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.438358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:32943 (local address 127.11.106.126:42625)
0619 07:00:41.438546 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.438550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.439475 (+ 925us) server_negotiation.cc:407] Connection header received
0619 07:00:41.441235 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.441241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.441307 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.441367 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:41.443106 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.443643 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.445163 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.445406 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.445545 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.448769 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:41.448774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:41.448777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:41.448811 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:41.450660 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.451215 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.451218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.451220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.451308 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.451739 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.451742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.451743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.451956 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:41.452062 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.452627 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.452761 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 07:00:41.453944 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.438830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:32943)
0619 07:00:41.439325 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.439368 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:41.440952 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:41.441802 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:41.441814 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:41.442363 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:41.442940 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.442955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.443777 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.443783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.445041 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.445050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.445599 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.445603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.445788 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.446427 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:41.446444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:41.448477 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:41.450781 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:41.450791 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:41.450812 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:41.451100 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:41.451407 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:41.451411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:41.451412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:41.451635 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:41.452073 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:41.452082 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:41.452349 (+ 267us) client_negotiation.cc:769] Sending connection context
0619 07:00:41.452552 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 07:00:41.452935 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
I20260619 07:00:41.455492 26330 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42625
I20260619 07:00:41.455806 26330 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:41.456363 26330 heartbeater.cc:511] Master 127.11.106.126:42625 requested a full tablet report, sending...
I20260619 07:00:41.457540 26004 ts_manager.cc:194] Registered new tserver with Master: ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083)
I20260619 07:00:41.458132 26004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:32943
W20260619 07:00:41.555522 26333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:41.555817 26333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:41.555915 26333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:41.559967 26333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:41.560058 26333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:41.560144 26333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:00:41.565513 26333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:43069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=42729
--tserver_master_addrs=127.11.106.126:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:41.566578 26333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:41.567653 26333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:41.577103 26341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:41.577103 26338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:41.577119 26339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:41.577288 26333 server_base.cc:1034] running on GCE node
I20260619 07:00:41.577844 26333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:41.578548 26333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:41.579712 26333 hybrid_clock.cc:648] HybridClock initialized: now 1781852441579694 us; error 30 us; skew 500 ppm
Jun 19 07:00:41 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:41.582445 26333 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:00:41.583390 26333 webserver.cc:460] Webserver started at http://127.11.106.67:42729/ using document root <none> and password file <none>
I20260619 07:00:41.583952 26333 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:41.584025 26333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:41.588088 26333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 07:00:41.590170 26348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:41.591617 26333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:00:41.591738 26333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "adc686a0f205482baa6f8fe4f373cf7b"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6d6f8bcb457ea997286bceb9d691c87b"
server_key_iv: "5486c0a64e2bc305b91e430201754400"
server_key_version: "encryptionkey@0"
I20260619 07:00:41.592268 26333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:41.601436 26333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:41.604475 26333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:41.604688 26333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:41.605211 26333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:41.606186 26333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:41.606240 26333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:41.606281 26333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:41.606297 26333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:41.615936 26333 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:43069
I20260619 07:00:41.615978 26461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:43069 every 8 connection(s)
I20260619 07:00:41.617079 26333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:00:41.617841 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 26333
I20260619 07:00:41.627319 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.618717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.11.106.126:42625)
0619 07:00:41.618871 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.618876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.619177 (+ 301us) server_negotiation.cc:407] Connection header received
0619 07:00:41.619333 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.619337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.619395 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.619502 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:41.620472 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.621325 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.622260 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.622507 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.622633 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.624087 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:41.624091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:41.624094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:41.624119 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:41.625620 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.626066 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.626072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.626074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.626171 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.626464 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.626468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.626469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.626648 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:41.626861 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.627041 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.627153 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 07:00:41 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852441, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:41.631881 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.619309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:37599 (local address 127.11.106.126:42625)
0619 07:00:41.620573 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.620578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.620699 (+ 121us) server_negotiation.cc:407] Connection header received
0619 07:00:41.622242 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.622245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.622294 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.622379 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:41.623556 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.624064 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.625625 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.625845 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.625995 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.628606 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:41.628609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:41.628612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:41.628634 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:41.630031 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.630551 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.630555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.630556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.630635 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:41.630955 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:41.630958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:41.630959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:41.631127 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:41.631209 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.631659 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.631757 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":98,"threads_started":1}
I20260619 07:00:41.632844 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.619852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:37599)
0619 07:00:41.620447 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.620586 (+ 139us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:41.622030 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:41.622567 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:41.622574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:41.622942 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:41.623403 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.623425 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.624197 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.624201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.625492 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.625502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.626025 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.626031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.626208 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.626753 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:41.626769 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:41.628428 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:41.630136 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:41.630141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:41.630154 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:41.630397 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:41.630726 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:41.630728 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:41.630730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:41.630859 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:41.631219 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:41.631224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:41.631443 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 07:00:41.631596 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 07:00:41.631847 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":156,"threads_started":1}
I20260619 07:00:41.634110 26462 heartbeater.cc:347] Connected to a master server at 127.11.106.126:42625
I20260619 07:00:41.634354 26462 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:41.634840 26462 heartbeater.cc:511] Master 127.11.106.126:42625 requested a full tablet report, sending...
I20260619 07:00:41.635797 26004 ts_manager.cc:194] Registered new tserver with Master: adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069)
I20260619 07:00:41.636303 26004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:37599
I20260619 07:00:41.641057 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:00:41.645103 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.641728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.11.106.126:42625)
0619 07:00:41.641843 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.641847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.641925 (+ 78us) server_negotiation.cc:407] Connection header received
0619 07:00:41.642053 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.642055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.642139 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.642217 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:00:41.642418 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.642909 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.643699 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.643921 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.644003 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.644350 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:00:41.644784 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:00:41.644874 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.644943 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.644992 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 07:00:41.645736 26004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:00:41.649013 26004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:00:41.657352 26120 tablet_service.cc:1467] Processing CreateTablet for tablet 198eada29d654243825616444cf8d2c8 (DEFAULT_TABLE table=test-table [id=09ae223693f84a63b24c231af869fa27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:41.657999 26120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198eada29d654243825616444cf8d2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:41.660743 26084 tablet_bootstrap.cc:492] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Bootstrap starting.
I20260619 07:00:41.661741 26084 tablet_bootstrap.cc:654] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:41.662051 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.656363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:46083 (local address 127.0.0.1:33188)
0619 07:00:41.656641 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.656654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:41.656745 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:41.657943 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:41.657945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:41.657962 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:41.658051 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.658058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.659449 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.659460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.660327 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.660338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.661579 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.661585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.661703 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.661739 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 07:00:41.661848 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 07:00:41.661922 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":29,"thread_start_us":132,"threads_started":1}
I20260619 07:00:41.662213 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.656871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.11.106.66:46083)
0619 07:00:41.657265 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.657278 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.657302 (+ 24us) server_negotiation.cc:407] Connection header received
0619 07:00:41.657402 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.657408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.657610 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.657780 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:41.658194 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.659318 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.660473 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.661346 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.661481 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.661752 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.661886 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.662008 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":186,"threads_started":1}
I20260619 07:00:41.663450 26084 tablet_bootstrap.cc:492] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: No bootstrap required, opened a new log
I20260619 07:00:41.663568 26084 ts_tablet_manager.cc:1397] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 07:00:41.664108 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.656094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43069 (local address 127.0.0.1:50060)
0619 07:00:41.656228 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.656243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:41.656323 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:41.657674 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:41.657677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:41.657698 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:41.657823 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.657833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.659336 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.659342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.660519 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.660530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.663689 (+ 3159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.663694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.663805 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.663833 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 07:00:41.663893 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 07:00:41.663975 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260619 07:00:41.664125 26084 raft_consensus.cc:348] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:41.664251 26084 raft_consensus.cc:374] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:41.664297 26084 raft_consensus.cc:729] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a105740decb74dd9b744333f46ae4a58, State: Initialized, Role: FOLLOWER
I20260619 07:00:41.664446 26084 consensus_queue.cc:260] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:41.664880 26084 ts_tablet_manager.cc:1428] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 07:00:41.664939 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.656335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.11.106.67:43069)
0619 07:00:41.657104 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.657124 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.657152 (+ 28us) server_negotiation.cc:407] Connection header received
0619 07:00:41.657241 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.657247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.657493 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.657692 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:41.657958 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.659176 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.662729 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.663524 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.664317 (+ 793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.664592 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.664656 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.664775 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":149,"threads_started":1}
I20260619 07:00:41.665308 26264 tablet_service.cc:1467] Processing CreateTablet for tablet 198eada29d654243825616444cf8d2c8 (DEFAULT_TABLE table=test-table [id=09ae223693f84a63b24c231af869fa27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:41.666529 26264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198eada29d654243825616444cf8d2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:41.667284 26396 tablet_service.cc:1467] Processing CreateTablet for tablet 198eada29d654243825616444cf8d2c8 (DEFAULT_TABLE table=test-table [id=09ae223693f84a63b24c231af869fa27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:41.668183 26396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198eada29d654243825616444cf8d2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:41.673507 26475 tablet_bootstrap.cc:492] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: Bootstrap starting.
I20260619 07:00:41.674733 26476 tablet_bootstrap.cc:492] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: Bootstrap starting.
I20260619 07:00:41.676143 26475 tablet_bootstrap.cc:654] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:41.677093 26475 log.cc:824] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:41.677218 26476 tablet_bootstrap.cc:654] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:41.678092 26476 log.cc:824] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:41.678972 26475 tablet_bootstrap.cc:492] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: No bootstrap required, opened a new log
I20260619 07:00:41.679180 26475 ts_tablet_manager.cc:1397] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 07:00:41.680320 26476 tablet_bootstrap.cc:492] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: No bootstrap required, opened a new log
I20260619 07:00:41.680581 26476 ts_tablet_manager.cc:1397] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 07:00:41.682570 26475 raft_consensus.cc:348] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:41.682783 26475 raft_consensus.cc:374] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:41.682849 26475 raft_consensus.cc:729] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae0e048281184851bc745fda2e8b3d85, State: Initialized, Role: FOLLOWER
I20260619 07:00:41.683246 26475 consensus_queue.cc:260] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:41.683914 26330 heartbeater.cc:503] Master 127.11.106.126:42625 was elected leader, sending a full tablet report...
I20260619 07:00:41.683809 26476 raft_consensus.cc:348] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:41.684073 26476 raft_consensus.cc:374] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:41.684139 26476 raft_consensus.cc:729] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc686a0f205482baa6f8fe4f373cf7b, State: Initialized, Role: FOLLOWER
I20260619 07:00:41.684159 26475 ts_tablet_manager.cc:1428] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:00:41.684707 26476 consensus_queue.cc:260] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:41.685346 26462 heartbeater.cc:503] Master 127.11.106.126:42625 was elected leader, sending a full tablet report...
I20260619 07:00:41.685734 26476 ts_tablet_manager.cc:1428] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260619 07:00:41.687389 26331 tablet.cc:2369] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 07:00:41.725822 26193 tablet.cc:2369] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:00:41.812376 26479 raft_consensus.cc:482] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:41.812666 26479 raft_consensus.cc:504] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:41.813908 26479 leader_election.cc:288] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069), a105740decb74dd9b744333f46ae4a58 (127.11.106.65:45663)
I20260619 07:00:41.817626 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.814214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43069 (local address 127.11.106.66:38833)
0619 07:00:41.814351 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.814367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:41.814482 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:41.814768 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:41.814771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:41.814790 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:41.814914 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.814919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.815684 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.815689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.816452 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.816461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.817177 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.817181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.817287 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.817311 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 07:00:41.817400 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 07:00:41.817489 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 07:00:41.817626 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.814347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:38833 (local address 127.11.106.67:43069)
0619 07:00:41.814479 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.814486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.814506 (+ 20us) server_negotiation.cc:407] Connection header received
0619 07:00:41.814575 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.814579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.814636 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.814754 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:41.815037 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.815559 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.816646 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.817048 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.817181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.817238 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.817420 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.817490 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 07:00:41.817938 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.814214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45663 (local address 127.11.106.66:38531)
0619 07:00:41.814728 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.814739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:41.814854 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:41.815148 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:41.815151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:41.815173 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:41.815294 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.815300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.816021 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.816026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.816874 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:41.816883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.817550 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.817554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:41.817640 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.817662 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 07:00:41.817729 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 07:00:41.817813 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":178,"thread_start_us":117,"threads_started":1}
I20260619 07:00:41.818321 26481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.814336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:38531 (local address 127.11.106.65:45663)
0619 07:00:41.814621 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.814627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.814826 (+ 199us) server_negotiation.cc:407] Connection header received
0619 07:00:41.814958 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.814963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.815024 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.815138 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:41.815410 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.815910 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.816994 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.817427 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.817558 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.818043 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.818087 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.818171 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260619 07:00:41.818346 26416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198eada29d654243825616444cf8d2c8" candidate_uuid: "ae0e048281184851bc745fda2e8b3d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc686a0f205482baa6f8fe4f373cf7b" is_pre_election: true
I20260619 07:00:41.818713 26416 raft_consensus.cc:2454] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0e048281184851bc745fda2e8b3d85 in term 0.
I20260619 07:00:41.818748 26128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198eada29d654243825616444cf8d2c8" candidate_uuid: "ae0e048281184851bc745fda2e8b3d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a105740decb74dd9b744333f46ae4a58" is_pre_election: true
I20260619 07:00:41.819022 26128 raft_consensus.cc:2454] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae0e048281184851bc745fda2e8b3d85 in term 0.
I20260619 07:00:41.819265 26217 leader_election.cc:302] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc686a0f205482baa6f8fe4f373cf7b, ae0e048281184851bc745fda2e8b3d85; no voters:
I20260619 07:00:41.819532 26479 raft_consensus.cc:2781] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:00:41.819662 26479 raft_consensus.cc:482] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:00:41.819711 26479 raft_consensus.cc:3037] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:41.820866 26479 raft_consensus.cc:504] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:41.821187 26479 leader_election.cc:288] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [CANDIDATE]: Term 1 election: Requested vote from peers adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069), a105740decb74dd9b744333f46ae4a58 (127.11.106.65:45663)
I20260619 07:00:41.821503 26416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198eada29d654243825616444cf8d2c8" candidate_uuid: "ae0e048281184851bc745fda2e8b3d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc686a0f205482baa6f8fe4f373cf7b"
I20260619 07:00:41.821522 26128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198eada29d654243825616444cf8d2c8" candidate_uuid: "ae0e048281184851bc745fda2e8b3d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a105740decb74dd9b744333f46ae4a58"
I20260619 07:00:41.821617 26416 raft_consensus.cc:3037] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:41.821655 26128 raft_consensus.cc:3037] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:41.822626 26128 raft_consensus.cc:2454] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0e048281184851bc745fda2e8b3d85 in term 1.
I20260619 07:00:41.822631 26416 raft_consensus.cc:2454] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae0e048281184851bc745fda2e8b3d85 in term 1.
I20260619 07:00:41.822932 26220 leader_election.cc:302] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a105740decb74dd9b744333f46ae4a58, ae0e048281184851bc745fda2e8b3d85; no voters:
I20260619 07:00:41.823115 26479 raft_consensus.cc:2781] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:00:41.823370 26479 raft_consensus.cc:686] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 1 LEADER]: Becoming Leader. State: Replica: ae0e048281184851bc745fda2e8b3d85, State: Running, Role: LEADER
I20260619 07:00:41.823731 26479 consensus_queue.cc:237] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:41.826920 26003 catalog_manager.cc:5495] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae0e048281184851bc745fda2e8b3d85 (127.11.106.66). New cstate: current_term: 1 leader_uuid: "ae0e048281184851bc745fda2e8b3d85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } health_report { overall_health: UNKNOWN } } }
I20260619 07:00:41.864980 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.861525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.11.106.66:46083)
0619 07:00:41.861690 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:41.861693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:41.861708 (+ 15us) server_negotiation.cc:407] Connection header received
0619 07:00:41.861786 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:41.861788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:41.861845 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:41.861950 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:00:41.862145 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.862682 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.863463 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:41.863692 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:41.863789 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:41.864081 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:00:41.864606 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:00:41.864729 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:41.864784 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 07:00:41.864840 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 07:00:41.868083 26463 tablet.cc:2369] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:00:41.871064 26416 raft_consensus.cc:1261] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [term 1 FOLLOWER]: Refusing update from remote peer ae0e048281184851bc745fda2e8b3d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:00:41.871069 26128 raft_consensus.cc:1261] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Refusing update from remote peer ae0e048281184851bc745fda2e8b3d85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:00:41.871776 26479 consensus_queue.cc:1035] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:00:41.871974 26483 consensus_queue.cc:1035] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:00:41.888064 26002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:00:41.888259 26002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:00:41.890708 26002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Sending DeleteTablet for 3 replicas of tablet 198eada29d654243825616444cf8d2c8
I20260619 07:00:41.891430 26396 tablet_service.cc:1514] Processing DeleteTablet for tablet 198eada29d654243825616444cf8d2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:41 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:50060
I20260619 07:00:41.891440 26264 tablet_service.cc:1514] Processing DeleteTablet for tablet 198eada29d654243825616444cf8d2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:41 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:33188
I20260619 07:00:41.891628 26120 tablet_service.cc:1514] Processing DeleteTablet for tablet 198eada29d654243825616444cf8d2c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:00:41 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:40274
I20260619 07:00:41.891917 26495 tablet_replica.cc:330] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: stopping tablet replica
I20260619 07:00:41.891948 26497 tablet_replica.cc:330] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: stopping tablet replica
I20260619 07:00:41.892081 26497 raft_consensus.cc:2229] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:00:41.892222 26497 raft_consensus.cc:2258] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:00:41.892247 26495 raft_consensus.cc:2229] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 1 LEADER]: Raft consensus shutting down.
I20260619 07:00:41.892417 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 26042
I20260619 07:00:41.893600 26495 raft_consensus.cc:2258] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:00:41.895005 26495 ts_tablet_manager.cc:1905] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:00:41.897837 26495 ts_tablet_manager.cc:1918] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:00:41.897926 26495 log.cc:1196] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/198eada29d654243825616444cf8d2c8
I20260619 07:00:41.898278 26495 ts_tablet_manager.cc:1939] T 198eada29d654243825616444cf8d2c8 P ae0e048281184851bc745fda2e8b3d85: Deleting consensus metadata
W20260619 07:00:41.898972 25991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:45663 (error 108)
I20260619 07:00:41.899225 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 26196
I20260619 07:00:41.900525 26496 tablet_replica.cc:330] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: stopping tablet replica
I20260619 07:00:41.900938 26496 raft_consensus.cc:2229] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:00:41.901281 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.900944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:45663 (local address 127.0.0.1:40286)
0619 07:00:41.901128 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.901196 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 07:00:41.901289 26496 raft_consensus.cc:2258] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 07:00:41.901449 25991 catalog_manager.cc:4584] TS a105740decb74dd9b744333f46ae4a58 (127.11.106.65:45663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 198eada29d654243825616444cf8d2c8: Network error: Client connection negotiation failed: client connection to 127.11.106.65:45663: connect: Connection refused (error 111)
I20260619 07:00:41.902678 26496 ts_tablet_manager.cc:1905] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:00:41.904837 26496 ts_tablet_manager.cc:1918] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:00:41.904958 26496 log.cc:1196] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/198eada29d654243825616444cf8d2c8
I20260619 07:00:41.905277 26496 ts_tablet_manager.cc:1939] T 198eada29d654243825616444cf8d2c8 P adc686a0f205482baa6f8fe4f373cf7b: Deleting consensus metadata
I20260619 07:00:41.906391 25988 catalog_manager.cc:4841] TS adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069): tablet 198eada29d654243825616444cf8d2c8 (table test-table [id=09ae223693f84a63b24c231af869fa27]) successfully deleted
I20260619 07:00:41.906545 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 26333
I20260619 07:00:41.908686 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:41.908327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:46083 (local address 127.0.0.1:33208)
0619 07:00:41.908455 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:41.908503 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.66:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 07:00:41.912573 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 25971
I20260619 07:00:41.918689 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42625
--webserver_interface=127.11.106.126
--webserver_port=43075
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:42.029376 26499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:42.029695 26499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:42.029769 26499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:42.033886 26499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:42.033979 26499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:42.034018 26499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:42.034080 26499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:42.034117 26499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:42.039551 26499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=43075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:42.040838 26499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:42.041839 26499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:42.050654 26505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:42.050776 26504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:42.050654 26507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:42.051039 26499 server_base.cc:1034] running on GCE node
I20260619 07:00:42.051855 26499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:42.053367 26499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:42.054617 26499 hybrid_clock.cc:648] HybridClock initialized: now 1781852442054597 us; error 32 us; skew 500 ppm
Jun 19 07:00:42 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:42.057685 26499 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:00:42.058589 26499 webserver.cc:460] Webserver started at http://127.11.106.126:43075/ using document root <none> and password file <none>
I20260619 07:00:42.059124 26499 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:42.059185 26499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:42.063077 26499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:00:42.065336 26514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:42.066695 26499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:00:42.066812 26499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "745ac748a8454c959c5940543d72902e"
format_stamp: "Formatted at 2026-06-19 07:00:39 on dist-test-slave-10d0"
server_key: "de6ce5f977fd1996f39a6857d63ccf8a"
server_key_iv: "a8a903abdcd33329bdfc4176d1ee8d67"
server_key_version: "encryptionkey@0"
I20260619 07:00:42.067368 26499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:42.089239 26499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:42.092183 26499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:42.092346 26499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:42.099498 26499 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:42625
I20260619 07:00:42.099835 26566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:42625 every 8 connection(s)
I20260619 07:00:42.100981 26499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:00:42.105098 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 26499
I20260619 07:00:42.105567 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:45663
--local_ip_for_outbound_sockets=127.11.106.65
--tserver_master_addrs=127.11.106.126:42625
--webserver_port=39169
--webserver_interface=127.11.106.65
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:00:42.105827 26567 sys_catalog.cc:263] Verifying existing consensus state
I20260619 07:00:42.108021 26567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Bootstrap starting.
I20260619 07:00:42.127372 26567 log.cc:824] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:42.138638 26567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 07:00:42.139038 26567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e: Bootstrap complete.
I20260619 07:00:42.143261 26567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:42.143651 26567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745ac748a8454c959c5940543d72902e, State: Initialized, Role: FOLLOWER
I20260619 07:00:42.144203 26567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:42.144325 26567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:00:42.144411 26567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:00:42.144537 26567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 2 FOLLOWER]: Advancing to term 3
I20260619 07:00:42.146624 26567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:42.146989 26567 leader_election.cc:302] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 745ac748a8454c959c5940543d72902e; no voters:
I20260619 07:00:42.147360 26567 leader_election.cc:288] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 07:00:42.147449 26571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 3 FOLLOWER]: Leader election won for term 3
I20260619 07:00:42.147645 26571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [term 3 LEADER]: Becoming Leader. State: Replica: 745ac748a8454c959c5940543d72902e, State: Running, Role: LEADER
I20260619 07:00:42.147951 26571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } }
I20260619 07:00:42.148694 26567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:42.148941 26571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 745ac748a8454c959c5940543d72902e. Latest consensus state: current_term: 3 leader_uuid: "745ac748a8454c959c5940543d72902e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } } }
I20260619 07:00:42.149052 26571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:42.149487 26575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:00:42.149448 26572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "745ac748a8454c959c5940543d72902e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "745ac748a8454c959c5940543d72902e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42625 } } }
I20260619 07:00:42.149667 26572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 745ac748a8454c959c5940543d72902e [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:42.155068 26575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:00:42.155784 26575 catalog_manager.cc:1259] Loaded cluster ID: e3e37d674837441c93dda339226b7db2
I20260619 07:00:42.155853 26575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 07:00:42.157006 26575 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4a0fcf19ad google::LogMessage::Fail() at ??:0
@ 0x7f4a0fcf488a google::LogMessage::SendToLog() at ??:0
@ 0x7f4a0fcf14de google::LogMessage::Flush() at ??:0
@ 0x7f4a0fcf2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4a1a630fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4a1a631d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4a1a62e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4a1a672640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4a11f433d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4a11f43c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4a11f453f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4a11f34512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4a135786db start_thread at ??:0
@ 0x7f4a0e7ef61f clone at ??:0
W20260619 07:00:42.240437 26569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:42.240819 26569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:42.240942 26569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:42.245820 26569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:42.245909 26569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:42.246026 26569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:00:42.252460 26569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:45663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=39169
--tserver_master_addrs=127.11.106.126:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:42.253708 26569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:42.254822 26569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:42.265187 26609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:42.265414 26611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:42.265237 26608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:42.265705 26569 server_base.cc:1034] running on GCE node
I20260619 07:00:42.266139 26569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:42.266849 26569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:42.268049 26569 hybrid_clock.cc:648] HybridClock initialized: now 1781852442268019 us; error 50 us; skew 500 ppm
Jun 19 07:00:42 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:42.271032 26569 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:00:42.272012 26569 webserver.cc:460] Webserver started at http://127.11.106.65:39169/ using document root <none> and password file <none>
I20260619 07:00:42.272771 26569 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:42.272841 26569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:42.276968 26569 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 07:00:42.279716 26618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:42.281131 26569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:00:42.281226 26569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "a105740decb74dd9b744333f46ae4a58"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6682879e2995578fdd03035f43532d73"
server_key_iv: "753cbbe3f23fcac02ceeec67674480bb"
server_key_version: "encryptionkey@0"
I20260619 07:00:42.281823 26569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:42.298108 26569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:42.301402 26569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:42.301610 26569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:42.302240 26569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:42.303436 26625 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 07:00:42.308712 26569 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 07:00:42.308799 26569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 07:00:42.308890 26569 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 07:00:42.310648 26569 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 07:00:42.310721 26569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 07:00:42.311022 26625 tablet_bootstrap.cc:492] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Bootstrap starting.
I20260619 07:00:42.322037 26569 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:45663
I20260619 07:00:42.322204 26732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:45663 every 8 connection(s)
I20260619 07:00:42.323446 26569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:00:42.332957 26625 log.cc:824] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:42.333454 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 26569
I20260619 07:00:42.333964 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:46083
--local_ip_for_outbound_sockets=127.11.106.66
--tserver_master_addrs=127.11.106.126:42625
--webserver_port=37481
--webserver_interface=127.11.106.66
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:00:42.342284 26625 tablet_bootstrap.cc:492] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 07:00:42.342732 26625 tablet_bootstrap.cc:492] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Bootstrap complete.
I20260619 07:00:42.344192 26625 ts_tablet_manager.cc:1397] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260619 07:00:42.347760 26625 raft_consensus.cc:348] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:42.348309 26625 raft_consensus.cc:729] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a105740decb74dd9b744333f46ae4a58, State: Initialized, Role: FOLLOWER
I20260619 07:00:42.348891 26625 consensus_queue.cc:260] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:42.350025 26625 ts_tablet_manager.cc:1428] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260619 07:00:42.421039 26735 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:00:42.325740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:52891)
0619 07:00:42.326239 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:42.326300 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:42.327152 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:42.420126 (+ 92974us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":139,"threads_started":1}
W20260619 07:00:42.421598 26733 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:42625 (0 consecutive failures): Network error: Failed to ping master at 127.11.106.126:42625: Client connection negotiation failed: client connection to 127.11.106.126:42625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 07:00:42.422418 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:42.422156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:36115)
0619 07:00:42.422302 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:42.422366 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 07:00:42.423157 26735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:42.422978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:50405)
0619 07:00:42.423073 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:42.423114 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 07:00:42.423384 26733 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 07:00:42.450986 26737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:42.451373 26737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:42.451454 26737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:42.456341 26737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:42.456429 26737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:42.456529 26737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:00:42.461905 26737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:46083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=37481
--tserver_master_addrs=127.11.106.126:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:42.462980 26737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:42.464006 26737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:42.473464 26744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:42.473469 26743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:42.473459 26746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:42.473595 26737 server_base.cc:1034] running on GCE node
I20260619 07:00:42.474121 26737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:42.474773 26737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:42.475946 26737 hybrid_clock.cc:648] HybridClock initialized: now 1781852442475931 us; error 36 us; skew 500 ppm
Jun 19 07:00:42 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:42.478725 26737 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:00:42.479641 26737 webserver.cc:460] Webserver started at http://127.11.106.66:37481/ using document root <none> and password file <none>
I20260619 07:00:42.480238 26737 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:42.480333 26737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:42.484396 26737 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:00:42.486868 26753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:42.488150 26737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:00:42.488284 26737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "ae0e048281184851bc745fda2e8b3d85"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "0772ae393e6279154644c47e523406a3"
server_key_iv: "48b28b33b3ccede80bdcad468e041e8f"
server_key_version: "encryptionkey@0"
I20260619 07:00:42.488909 26737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:42.509649 26737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:42.512852 26737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:42.513074 26737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:42.513734 26737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:42.514600 26737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:42.514680 26737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:42.514765 26737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:42.514801 26737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:42.523883 26737 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:46083
I20260619 07:00:42.523907 26866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:46083 every 8 connection(s)
I20260619 07:00:42.524957 26737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:00:42.528892 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:42.527155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:51319)
0619 07:00:42.527689 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:42.528120 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":103,"threads_started":1}
W20260619 07:00:42.529376 26867 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:42625 (0 consecutive failures): Network error: Failed to ping master at 127.11.106.126:42625: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
I20260619 07:00:42.529978 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:42.529771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:48647)
0619 07:00:42.529876 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:42.529923 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 07:00:42.530735 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:42.530458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:51475)
0619 07:00:42.530611 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:42.530670 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 07:00:42.530790 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 26737
W20260619 07:00:42.530952 26867 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 07:00:42.531235 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:43069
--local_ip_for_outbound_sockets=127.11.106.67
--tserver_master_addrs=127.11.106.126:42625
--webserver_port=42729
--webserver_interface=127.11.106.67
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:42.642278 26870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:42.642535 26870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:42.642578 26870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:42.646787 26870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:42.646852 26870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:42.646940 26870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:00:42.652459 26870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:43069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=42729
--tserver_master_addrs=127.11.106.126:42625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:42.653599 26870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:42.654659 26870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:42.665076 26878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:42.665114 26870 server_base.cc:1034] running on GCE node
W20260619 07:00:42.665081 26875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:42.665076 26876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:42.665968 26870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:42.666702 26870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:42.667869 26870 hybrid_clock.cc:648] HybridClock initialized: now 1781852442667844 us; error 37 us; skew 500 ppm
Jun 19 07:00:42 dist-test-slave-10d0 krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:42.670907 26870 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:00:42.671821 26870 webserver.cc:460] Webserver started at http://127.11.106.67:42729/ using document root <none> and password file <none>
I20260619 07:00:42.672372 26870 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:42.672432 26870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:42.676357 26870 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 07:00:42.678996 26885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:42.680392 26870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 07:00:42.680517 26870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "adc686a0f205482baa6f8fe4f373cf7b"
format_stamp: "Formatted at 2026-06-19 07:00:40 on dist-test-slave-10d0"
server_key: "6d6f8bcb457ea997286bceb9d691c87b"
server_key_iv: "5486c0a64e2bc305b91e430201754400"
server_key_version: "encryptionkey@0"
I20260619 07:00:42.681083 26870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:42.694629 26870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:42.697796 26870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:42.698010 26870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:42.698673 26870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:42.699518 26870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:42.699563 26870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:42.699630 26870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:42.699662 26870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:42.709206 26870 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:43069
I20260619 07:00:42.709244 26998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:43069 every 8 connection(s)
I20260619 07:00:42.710273 26870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:00:42.714584 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:42.712872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:35713)
0619 07:00:42.713401 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:42.713840 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":112,"threads_started":1}
W20260619 07:00:42.715116 26999 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:42625 (0 consecutive failures): Network error: Failed to ping master at 127.11.106.126:42625: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
I20260619 07:00:42.715721 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:42.715486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:58099)
0619 07:00:42.715617 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:42.715666 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 07:00:42.716337 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:42.716154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:37733)
0619 07:00:42.716248 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:42.716289 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 07:00:42.716521 26999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 07:00:42.717929 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 26870
I20260619 07:00:43.424383 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:43.423978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:40717)
0619 07:00:43.424219 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:43.424272 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260619 07:00:43.531934 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:43.531541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:58993)
0619 07:00:43.531784 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:43.531839 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
I20260619 07:00:43.610728 27005 raft_consensus.cc:482] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:43.610965 27005 raft_consensus.cc:504] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:43.612135 27005 leader_election.cc:288] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069), ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083)
Jun 19 07:00:43 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.66@KRBTEST.COM
Jun 19 07:00:43 dist-test-slave-10d0 krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852442, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.67@KRBTEST.COM
I20260619 07:00:43.625887 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:43.612310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43069 (local address 127.11.106.65:38753)
0619 07:00:43.612432 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:43.612450 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:43.612595 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:43.614578 (+ 1983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:43.614584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:43.615086 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:43.615580 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:43.615594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:43.616944 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:43.616946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:43.617789 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:43.617798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:43.618495 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:43.618498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:43.618732 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:43.619362 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:43.619377 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:43.621277 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:43.623431 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:43.623437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:43.623440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:43.623694 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:43.624028 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:43.624032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:43.624034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:43.624138 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:43.625046 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:43.625048 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:43.625188 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 07:00:43.625373 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 07:00:43.625687 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":184,"spinlock_wait_cycles":3840}
I20260619 07:00:43.625900 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:43.612687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:54799 (local address 127.11.106.66:46083)
0619 07:00:43.613001 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:43.613018 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:43.613047 (+ 29us) server_negotiation.cc:407] Connection header received
0619 07:00:43.613804 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:43.613810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:43.614240 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:43.614543 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:43.615766 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:43.616756 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:43.617921 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:43.618373 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:43.618541 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:43.621113 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:43.621118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:43.621129 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:43.621154 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:43.623222 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:43.623761 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:43.623767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:43.623772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:43.623887 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:43.624237 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:43.624240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:43.624242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:43.624767 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:43.624925 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:43.625485 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 07:00:43.625736 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":64,"threads_started":1}
I20260619 07:00:43.625943 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:43.612499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:38753 (local address 127.11.106.67:43069)
0619 07:00:43.612891 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:43.612908 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:43.612961 (+ 53us) server_negotiation.cc:407] Connection header received
0619 07:00:43.613709 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:43.613720 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:43.614200 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:43.614508 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:43.615759 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:43.616788 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:43.617912 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:43.618374 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:43.618551 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:43.621465 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:43.621470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:43.621480 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:43.621504 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:43.623319 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:43.623803 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:43.623809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:43.623821 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:43.623924 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:43.624428 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:43.624431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:43.624432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:43.624951 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:43.625072 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:43.625480 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 07:00:43.625758 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260619 07:00:43.625932 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:43.612397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:46083 (local address 127.11.106.65:54799)
0619 07:00:43.612767 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:43.612782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:43.612896 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:43.614577 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:43.614584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:43.615087 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:43.615582 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:43.615594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:43.616886 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:43.616891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:43.617789 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:43.617799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:43.618493 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:43.618497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:43.618750 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:43.619416 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:43.619429 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:43.620899 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:43.623345 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:43.623351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:43.623364 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:43.623646 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:43.623993 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:43.623996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:43.623998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:43.624134 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:43.624922 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:43.624928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:43.625196 (+ 268us) client_negotiation.cc:769] Sending connection context
0619 07:00:43.625373 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 07:00:43.625774 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":258,"spinlock_wait_cycles":32896,"thread_start_us":63,"threads_started":1}
W20260619 07:00:43.628747 26621 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
W20260619 07:00:43.629395 26619 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
I20260619 07:00:43.629479 26619 leader_election.cc:302] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a105740decb74dd9b744333f46ae4a58; no voters: adc686a0f205482baa6f8fe4f373cf7b, ae0e048281184851bc745fda2e8b3d85
I20260619 07:00:43.629673 27005 raft_consensus.cc:2726] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:00:43.717396 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:43.717024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:35517)
0619 07:00:43.717258 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:43.717308 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260619 07:00:44.425740 27010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:44.425286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:44981)
0619 07:00:44.425568 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:44.425630 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
I20260619 07:00:44.533206 27011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:44.532803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:35657)
0619 07:00:44.533049 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:44.533105 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260619 07:00:44.718626 27012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:44.718242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:48151)
0619 07:00:44.718470 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:44.718523 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260619 07:00:45.427009 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:45.426550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:38305)
0619 07:00:45.426832 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:45.426897 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
I20260619 07:00:45.523932 27014 raft_consensus.cc:482] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:45.524129 27014 raft_consensus.cc:504] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:45.524626 27014 leader_election.cc:288] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069), ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083)
W20260619 07:00:45.525465 26621 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
W20260619 07:00:45.525574 26619 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
I20260619 07:00:45.525672 26619 leader_election.cc:302] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a105740decb74dd9b744333f46ae4a58; no voters: adc686a0f205482baa6f8fe4f373cf7b, ae0e048281184851bc745fda2e8b3d85
I20260619 07:00:45.525810 27014 raft_consensus.cc:2726] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:00:45.534403 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:45.534015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:40209)
0619 07:00:45.534255 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:45.534307 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260619 07:00:45.719985 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:45.719532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:59255)
0619 07:00:45.719811 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:45.719876 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260619 07:00:46.428385 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:46.427943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:39959)
0619 07:00:46.428227 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:46.428287 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":107,"threads_started":1}
I20260619 07:00:46.535693 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:46.535254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:60091)
0619 07:00:46.535514 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:46.535600 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260619 07:00:46.721314 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:46.720879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:33041)
0619 07:00:46.721147 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:46.721212 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260619 07:00:47.430045 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:47.429493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:48823)
0619 07:00:47.429837 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:47.429906 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":80,"threads_started":1}
I20260619 07:00:47.537060 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:47.536623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:46483)
0619 07:00:47.536903 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:47.536967 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260619 07:00:47.722903 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:47.722403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:55799)
0619 07:00:47.722733 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:47.722806 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260619 07:00:48.431623 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:48.431126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:35043)
0619 07:00:48.431441 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:48.431523 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260619 07:00:48.538301 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:48.537854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:60797)
0619 07:00:48.538124 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:48.538205 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260619 07:00:48.724005 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:48.723579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:56869)
0619 07:00:48.723851 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:48.723906 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260619 07:00:48.866191 27026 raft_consensus.cc:482] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:48.866382 27026 raft_consensus.cc:504] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:48.866796 27026 leader_election.cc:288] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069), ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083)
W20260619 07:00:48.867507 26621 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
W20260619 07:00:48.867654 26619 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
I20260619 07:00:48.867749 26619 leader_election.cc:302] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a105740decb74dd9b744333f46ae4a58; no voters: adc686a0f205482baa6f8fe4f373cf7b, ae0e048281184851bc745fda2e8b3d85
I20260619 07:00:48.867906 27026 raft_consensus.cc:2726] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:00:49.433022 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:49.432537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:35995)
0619 07:00:49.432868 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:49.432925 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":69,"threads_started":1}
I20260619 07:00:49.539500 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:49.539050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:50557)
0619 07:00:49.539342 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:49.539404 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
I20260619 07:00:49.725186 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:49.724738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:36839)
0619 07:00:49.725031 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:49.725089 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":69,"threads_started":1}
I20260619 07:00:50.434104 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:50.433736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:58165)
0619 07:00:50.433959 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:50.434011 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260619 07:00:50.540736 27031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:50.540253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:40955)
0619 07:00:50.540525 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:50.540614 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
I20260619 07:00:50.726256 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:50.725879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:53719)
0619 07:00:50.726104 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:50.726159 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260619 07:00:51.435329 27033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:51.434863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:36549)
0619 07:00:51.435170 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:51.435231 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":78,"threads_started":1}
I20260619 07:00:51.541857 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:51.541444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:57399)
0619 07:00:51.541704 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:51.541761 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260619 07:00:51.727538 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:51.727120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:40947)
0619 07:00:51.727378 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:51.727432 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
I20260619 07:00:52.420830 27036 raft_consensus.cc:482] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:52.420957 27036 raft_consensus.cc:504] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:52.421343 27036 leader_election.cc:288] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069), ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083)
W20260619 07:00:52.422142 26621 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
W20260619 07:00:52.422245 26619 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
I20260619 07:00:52.422353 26619 leader_election.cc:302] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a105740decb74dd9b744333f46ae4a58; no voters: adc686a0f205482baa6f8fe4f373cf7b, ae0e048281184851bc745fda2e8b3d85
I20260619 07:00:52.422509 27036 raft_consensus.cc:2726] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:00:52.436687 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:52.436210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:38849)
0619 07:00:52.436471 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:52.436567 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":65,"threads_started":1}
I20260619 07:00:52.543136 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:52.542704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:33361)
0619 07:00:52.542975 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:52.543046 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
I20260619 07:00:52.728722 27039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:52.728276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:53737)
0619 07:00:52.728505 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:52.728617 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260619 07:00:53.437834 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:53.437417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:58787)
0619 07:00:53.437682 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:53.437739 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260619 07:00:53.544292 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:53.543874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:57197)
0619 07:00:53.544140 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:53.544198 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":71,"threads_started":1}
I20260619 07:00:53.729848 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:53.729447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:40349)
0619 07:00:53.729698 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:53.729755 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
I20260619 07:00:54.439112 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:54.438702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:55943)
0619 07:00:54.438951 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:54.439014 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260619 07:00:54.545557 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:54.545106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:41487)
0619 07:00:54.545400 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:54.545459 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
I20260619 07:00:54.730981 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:54.730560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:41869)
0619 07:00:54.730824 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:54.730883 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260619 07:00:55.440388 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:55.439949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:35977)
0619 07:00:55.440238 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:55.440291 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
I20260619 07:00:55.546813 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:55.546286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:34531)
0619 07:00:55.546655 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:55.546719 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":94,"threads_started":1}
I20260619 07:00:55.732317 27048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:55.731867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:44739)
0619 07:00:55.732152 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:55.732220 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":117,"threads_started":1}
I20260619 07:00:56.340323 27049 raft_consensus.cc:482] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:56.340453 27049 raft_consensus.cc:504] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc686a0f205482baa6f8fe4f373cf7b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43069 } } peers { permanent_uuid: "ae0e048281184851bc745fda2e8b3d85" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 46083 } } peers { permanent_uuid: "a105740decb74dd9b744333f46ae4a58" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 45663 } }
I20260619 07:00:56.341024 27049 leader_election.cc:288] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069), ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083)
W20260619 07:00:56.341796 26619 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer adc686a0f205482baa6f8fe4f373cf7b (127.11.106.67:43069): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
W20260619 07:00:56.341889 26621 leader_election.cc:341] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae0e048281184851bc745fda2e8b3d85 (127.11.106.66:46083): Not found: Tablet not found: 198eada29d654243825616444cf8d2c8
I20260619 07:00:56.341951 26621 leader_election.cc:302] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a105740decb74dd9b744333f46ae4a58; no voters: adc686a0f205482baa6f8fe4f373cf7b, ae0e048281184851bc745fda2e8b3d85
I20260619 07:00:56.342082 27049 raft_consensus.cc:2726] T 198eada29d654243825616444cf8d2c8 P a105740decb74dd9b744333f46ae4a58 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:00:56.441565 27050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:56.441113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:45341)
0619 07:00:56.441413 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:56.441469 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
I20260619 07:00:56.548074 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:56.547657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:37099)
0619 07:00:56.547924 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:56.547979 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260619 07:00:56.733485 27052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:56.733110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.67:44805)
0619 07:00:56.733339 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:56.733391 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260619 07:00:57.442902 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:57.442493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.65:52081)
0619 07:00:57.442746 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:57.442807 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260619 07:00:57.549340 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:57.548973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:42625 (local address 127.11.106.66:44985)
0619 07:00:57.549200 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:57.549248 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260619 07:00:57.721060 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 26569
I20260619 07:00:57.726941 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 26737
I20260619 07:00:57.732303 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 26870
2026-06-19T07:00:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19339 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:00:57 dist-test-slave-10d0 krb5kdc[27057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:00:57 dist-test-slave-10d0 krb5kdc[27057](info): set up 2 sockets
Jun 19 07:00:57 dist-test-slave-10d0 krb5kdc[27057](info): commencing operation
krb5kdc: starting...
W20260619 07:00:58.781330 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:00:58 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:00:58.901682 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38655
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:38655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:59.013814 27070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:59.014096 27070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:59.014137 27070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:59.018430 27070 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 07:00:59.018491 27070 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 07:00:59.018507 27070 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 07:00:59.018522 27070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:00:59.018540 27070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:59.018603 27070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:59.018632 27070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:00:59.018662 27070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:00:59.023986 27070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:38655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:59.025107 27070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:59.026160 27070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 07:00:59.034991 27070 server_base.cc:1034] running on GCE node
W20260619 07:00:59.035179 27076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:59.035182 27078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:59.035315 27075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:59.035928 27070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:59.037130 27070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:59.038321 27070 hybrid_clock.cc:648] HybridClock initialized: now 1781852459038292 us; error 46 us; skew 500 ppm
Jun 19 07:00:59 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:59.041049 27070 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:00:59.041956 27070 webserver.cc:460] Webserver started at http://127.11.106.126:41631/ using document root <none> and password file <none>
I20260619 07:00:59.042555 27070 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:59.042649 27070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:59.042878 27070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:59.044978 27070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "93c478b22a0342a28a7152e2206fabc0"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "24eaee16d069da5a8985551b97854d74"
server_key_iv: "cbb7b53b3defcffef1aaa201a0eec871"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.045536 27070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "93c478b22a0342a28a7152e2206fabc0"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "24eaee16d069da5a8985551b97854d74"
server_key_iv: "cbb7b53b3defcffef1aaa201a0eec871"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.049226 27070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 07:00:59.051748 27085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.053028 27070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:00:59.053177 27070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "93c478b22a0342a28a7152e2206fabc0"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "24eaee16d069da5a8985551b97854d74"
server_key_iv: "cbb7b53b3defcffef1aaa201a0eec871"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.053306 27070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:59.063519 27070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:59.066250 27070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:59.066447 27070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:59.073684 27070 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:38655
I20260619 07:00:59.073947 27137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:38655 every 8 connection(s)
I20260619 07:00:59.074625 27070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:00:59.077378 27138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:59.078099 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 27070
I20260619 07:00:59.078248 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:00:59.078574 11689 external_mini_cluster.cc:1428] Setting key 0ec0c43cfa43f070a3af7f31bdaf675e
I20260619 07:00:59.083815 27138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Bootstrap starting.
I20260619 07:00:59.086445 27138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:59.087316 27138 log.cc:824] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Log is configured to *not* fsync() on all Append() calls
Jun 19 07:00:59 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:59.089149 27138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: No bootstrap required, opened a new log
I20260619 07:00:59.092494 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.079924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.11.106.126:38655)
0619 07:00:59.080343 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.080357 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.080393 (+ 36us) server_negotiation.cc:407] Connection header received
0619 07:00:59.081091 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.081107 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.081438 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.081713 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:59.082617 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.083572 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.084703 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.085155 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.085272 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.087996 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:59.088001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:59.088013 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:59.088040 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:59.089799 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.090246 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.090252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.090256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.090361 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.090669 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.090672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.090674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.091041 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:59.091269 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.091540 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.091812 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":147,"threads_started":1}
I20260619 07:00:59.092486 27138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:00:59.092778 27138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:59.092832 27138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c478b22a0342a28a7152e2206fabc0, State: Initialized, Role: FOLLOWER
I20260619 07:00:59.093289 27138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:00:59.093407 27138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:00:59.093487 27138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:00:59.093581 27138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:59.094628 27138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:00:59.094945 27138 leader_election.cc:302] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93c478b22a0342a28a7152e2206fabc0; no voters:
I20260619 07:00:59.095227 27138 leader_election.cc:288] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:00:59.095345 27143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:00:59.095590 27143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 1 LEADER]: Becoming Leader. State: Replica: 93c478b22a0342a28a7152e2206fabc0, State: Running, Role: LEADER
I20260619 07:00:59.095968 27143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:00:59.096403 27138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:00:59.097692 27144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93c478b22a0342a28a7152e2206fabc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } } }
I20260619 07:00:59.097733 27145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93c478b22a0342a28a7152e2206fabc0. Latest consensus state: current_term: 1 leader_uuid: "93c478b22a0342a28a7152e2206fabc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } } }
I20260619 07:00:59.097822 27144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:59.097832 27145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: This master's current role is: LEADER
I20260619 07:00:59.098268 27153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:00:59.101342 27153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:00:59.106757 27153 catalog_manager.cc:1347] Generated new cluster ID: bdbef68fcc5845d08e755308da02fd41
I20260619 07:00:59.106825 27153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:00:59.123864 27153 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:00:59.124851 27153 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:00:59.130760 27153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Generated new TSK 0
I20260619 07:00:59.131446 27153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 07:00:59.174919 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38655
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:59.292207 27164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:59.292461 27164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:59.292505 27164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:59.296487 27164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:59.296618 27164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:59.296716 27164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:00:59.302338 27164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:59.303401 27164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:59.304450 27164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:59.314182 27172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:59.314198 27170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:59.314203 27169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:59.314247 27164 server_base.cc:1034] running on GCE node
I20260619 07:00:59.314791 27164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:59.315423 27164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:59.316617 27164 hybrid_clock.cc:648] HybridClock initialized: now 1781852459316600 us; error 28 us; skew 500 ppm
Jun 19 07:00:59 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:59.319568 27164 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:00:59.320441 27164 webserver.cc:460] Webserver started at http://127.11.106.65:39103/ using document root <none> and password file <none>
I20260619 07:00:59.321060 27164 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:59.321151 27164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:59.321384 27164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:59.323400 27164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "72187e72c72a4c68897d702fb2e8c54b"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "b6d13ce2de93b182ed3152a05db721bf"
server_key_iv: "c5f5fca4e50432546a877b943c219373"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.323959 27164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "72187e72c72a4c68897d702fb2e8c54b"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "b6d13ce2de93b182ed3152a05db721bf"
server_key_iv: "c5f5fca4e50432546a877b943c219373"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.327682 27164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 07:00:59.330085 27179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.331224 27164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:00:59.331365 27164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "72187e72c72a4c68897d702fb2e8c54b"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "b6d13ce2de93b182ed3152a05db721bf"
server_key_iv: "c5f5fca4e50432546a877b943c219373"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.331473 27164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:59.347090 27164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:59.349938 27164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:59.350142 27164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:59.350713 27164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:59.351521 27164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:59.351591 27164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.351665 27164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:59.351714 27164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.361434 27164 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:38839
I20260619 07:00:59.361464 27292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:38839 every 8 connection(s)
I20260619 07:00:59.362296 27164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
Jun 19 07:00:59 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:59.372223 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 27164
I20260619 07:00:59.372341 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 07:00:59.372690 11689 external_mini_cluster.cc:1428] Setting key 9cfb16c8f4b99ba8c71b788a779d0b95
I20260619 07:00:59.376513 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.364054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:37401 (local address 127.11.106.126:38655)
0619 07:00:59.364200 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.364205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.365686 (+ 1481us) server_negotiation.cc:407] Connection header received
0619 07:00:59.366524 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.366527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.366572 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.366652 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:59.367841 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.368402 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.369623 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.369854 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.369928 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.372604 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:59.372607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:59.372609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:59.372635 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:59.374307 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.374825 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.374829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.374831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.374919 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.375270 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.375275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.375277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.375554 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:59.375658 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.376160 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.376301 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 07:00:59.377239 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.364515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:37401)
0619 07:00:59.365515 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:59.365561 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:59.366333 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:59.366829 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:59.366836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:59.367212 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:59.367698 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.367712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.368594 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.368602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.369501 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.369511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.370002 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.370005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.370156 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.370829 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:59.370846 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:59.372395 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:59.374426 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:59.374432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:59.374446 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:59.374717 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:59.375020 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:59.375023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:59.375025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:59.375172 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:59.375683 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:59.375688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:59.375915 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 07:00:59.376074 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 07:00:59.376331 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":536,"threads_started":1}
I20260619 07:00:59.378582 27293 heartbeater.cc:347] Connected to a master server at 127.11.106.126:38655
I20260619 07:00:59.378854 27293 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:59.379491 27293 heartbeater.cc:511] Master 127.11.106.126:38655 requested a full tablet report, sending...
I20260619 07:00:59.381163 27102 ts_manager.cc:194] Registered new tserver with Master: 72187e72c72a4c68897d702fb2e8c54b (127.11.106.65:38839)
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 07:00:59.382261 27102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:37401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 07:00:59.399989 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38655
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:59.513970 27298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:59.514223 27298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:59.514307 27298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:59.518862 27298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:59.518922 27298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:59.519021 27298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:00:59.524346 27298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:59.525516 27298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:59.526554 27298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:59.535970 27303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:59.535964 27304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:59.535961 27306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:59.536242 27298 server_base.cc:1034] running on GCE node
I20260619 07:00:59.536681 27298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:59.537321 27298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:59.538511 27298 hybrid_clock.cc:648] HybridClock initialized: now 1781852459538493 us; error 38 us; skew 500 ppm
Jun 19 07:00:59 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:59.541239 27298 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:00:59.542093 27298 webserver.cc:460] Webserver started at http://127.11.106.66:43571/ using document root <none> and password file <none>
I20260619 07:00:59.542668 27298 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:59.542760 27298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:59.542995 27298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:59.545143 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "87b0b40567eb76d085a220d6b485ce83"
server_key_iv: "ee74407e502732390ba32d708dd2e90e"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.545701 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "87b0b40567eb76d085a220d6b485ce83"
server_key_iv: "ee74407e502732390ba32d708dd2e90e"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.549764 27298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 07:00:59.552101 27313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.553331 27298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:00:59.553464 27298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "87b0b40567eb76d085a220d6b485ce83"
server_key_iv: "ee74407e502732390ba32d708dd2e90e"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.553575 27298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:59.577554 27298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:59.581015 27298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:59.581233 27298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:59.581846 27298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:59.582691 27298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:59.582760 27298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.582834 27298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:59.582887 27298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.592208 27298 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:41001
I20260619 07:00:59.592216 27426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:41001 every 8 connection(s)
I20260619 07:00:59.593185 27298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:00:59.596343 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 27298
I20260619 07:00:59.596488 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 07:00:59.596812 11689 external_mini_cluster.cc:1428] Setting key ad9a9e2f4dc15cfaaf880afc9eafe4a9
Jun 19 07:00:59 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 07:00:59.608007 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.595010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:59847 (local address 127.11.106.126:38655)
0619 07:00:59.595162 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.595165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.596177 (+ 1012us) server_negotiation.cc:407] Connection header received
0619 07:00:59.597348 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.597352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.597415 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.597493 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:59.598899 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.599417 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.601227 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.601440 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.601524 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.604402 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:59.604405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:59.604407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:59.604430 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:59.605907 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.606417 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.606422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.606424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.606537 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.606880 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.606885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.606887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.607069 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:59.607178 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.607722 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.607841 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 07:00:59.609082 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.595384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:59847)
0619 07:00:59.596021 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:59.596063 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:59.597091 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:59.597793 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:59.597800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:59.598260 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:59.598753 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.598767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.599547 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.599550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.601102 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.601116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.601597 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.601604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.601850 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.602550 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:59.602576 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:59.604231 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:59.606026 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:59.606032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:59.606048 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:59.606313 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:59.606631 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:59.606635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:59.606637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:59.606785 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:59.607179 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:59.607186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:59.607486 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 07:00:59.607681 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 07:00:59.608008 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":108,"threads_started":1}
I20260619 07:00:59.610260 27427 heartbeater.cc:347] Connected to a master server at 127.11.106.126:38655
I20260619 07:00:59.610525 27427 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:59.611063 27427 heartbeater.cc:511] Master 127.11.106.126:38655 requested a full tablet report, sending...
I20260619 07:00:59.612180 27102 ts_manager.cc:194] Registered new tserver with Master: b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001)
I20260619 07:00:59.612807 27102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:59847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 07:00:59.623914 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38655
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:00:59.735728 27432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:00:59.735988 27432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:00:59.736033 27432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:00:59.740272 27432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:00:59.740334 27432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:00:59.740396 27432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:00:59.745710 27432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:00:59.746752 27432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:00:59.747740 27432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:00:59.757630 27440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:59.757714 27438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:00:59.757714 27437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:00:59.757965 27432 server_base.cc:1034] running on GCE node
I20260619 07:00:59.758406 27432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:00:59.759140 27432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:00:59.760339 27432 hybrid_clock.cc:648] HybridClock initialized: now 1781852459760321 us; error 29 us; skew 500 ppm
Jun 19 07:00:59 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:00:59.763012 27432 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:00:59.763870 27432 webserver.cc:460] Webserver started at http://127.11.106.67:42227/ using document root <none> and password file <none>
I20260619 07:00:59.764427 27432 fs_manager.cc:359] Metadata directory not provided
I20260619 07:00:59.764518 27432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:00:59.764781 27432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:00:59.766878 27432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "36f2522d45ce41a9b29b5af9a802d0f9"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "7c5136418f4027a9e20750e8b8e407bc"
server_key_iv: "e3aaad4d06cae2dcffed9437566ad048"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.767441 27432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "36f2522d45ce41a9b29b5af9a802d0f9"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "7c5136418f4027a9e20750e8b8e407bc"
server_key_iv: "e3aaad4d06cae2dcffed9437566ad048"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.771056 27432 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 07:00:59.773468 27447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.774678 27432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:00:59.774806 27432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "36f2522d45ce41a9b29b5af9a802d0f9"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "7c5136418f4027a9e20750e8b8e407bc"
server_key_iv: "e3aaad4d06cae2dcffed9437566ad048"
server_key_version: "encryptionkey@0"
I20260619 07:00:59.774914 27432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:00:59.794047 27432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:00:59.797425 27432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:00:59.797636 27432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:00:59.798291 27432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:00:59.799147 27432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:00:59.799216 27432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.799291 27432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:00:59.799335 27432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:00:59.808028 27432 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:43955
I20260619 07:00:59.808066 27560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:43955 every 8 connection(s)
I20260619 07:00:59.808966 27432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:00:59.810081 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 27432
I20260619 07:00:59.810214 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 07:00:59.810530 11689 external_mini_cluster.cc:1428] Setting key 567b1c6ba56a0d83c82d7ac292ce2d96
Jun 19 07:00:59 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852459, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:00:59.822345 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.812319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:51297 (local address 127.11.106.126:38655)
0619 07:00:59.812460 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.812464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.812478 (+ 14us) server_negotiation.cc:407] Connection header received
0619 07:00:59.813351 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.813354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.813406 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.813487 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:59.814619 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.815152 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.816307 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.816529 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.816695 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.819053 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:59.819057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:59.819060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:59.819092 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:59.820285 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.820866 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.820871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.820873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.820958 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.821306 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.821311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.821313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.821488 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:59.821597 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.822057 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.822175 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 07:00:59.823163 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.811388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:51297)
0619 07:00:59.812169 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:59.812210 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:59.813153 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:59.813677 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:59.813685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:59.814024 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:00:59.814480 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.814493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.815277 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.815281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.816178 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.816188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.816728 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.816732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.816907 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.817470 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:00:59.817486 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:00:59.818895 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:00:59.820409 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:59.820414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:59.820428 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:59.820758 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:59.821057 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:00:59.821060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:00:59.821062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:00:59.821207 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:00:59.821596 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:00:59.821603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:00:59.821830 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 07:00:59.822015 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 07:00:59.822324 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":138,"threads_started":1}
I20260619 07:00:59.824298 27561 heartbeater.cc:347] Connected to a master server at 127.11.106.126:38655
I20260619 07:00:59.824615 27561 heartbeater.cc:464] Registering TS with master...
I20260619 07:00:59.825111 27561 heartbeater.cc:511] Master 127.11.106.126:38655 requested a full tablet report, sending...
I20260619 07:00:59.826087 27102 ts_manager.cc:194] Registered new tserver with Master: 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955)
I20260619 07:00:59.826650 27102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:51297
I20260619 07:00:59.836321 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:00:59.845044 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.838260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.11.106.126:38655)
0619 07:00:59.838356 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.838360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.838581 (+ 221us) server_negotiation.cc:407] Connection header received
0619 07:00:59.838677 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.838680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.838722 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.838784 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:00:59.839404 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.839897 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.840699 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.840903 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.840979 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.842243 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:00:59.842245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:00:59.842247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:00:59.842264 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:00:59.843454 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.843830 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.843832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.843833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.843903 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:00:59.844161 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:00:59.844164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:00:59.844165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:00:59.844363 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:00:59.844462 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.844728 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.844819 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 07:00:59.847832 27102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:00:59.850900 27102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:00:59.865664 27574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.860263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43955 (local address 127.0.0.1:44966)
0619 07:00:59.860499 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:59.860514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:59.860619 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:59.861302 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:59.861305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:59.861327 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:59.861415 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.861421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.862820 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.862824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.863967 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.863978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.865091 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.865094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.865207 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.865256 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 07:00:59.865356 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 07:00:59.865506 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":116,"thread_start_us":93,"threads_started":1}
I20260619 07:00:59.865670 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.859587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.11.106.66:41001)
0619 07:00:59.859997 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.860009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.860079 (+ 70us) server_negotiation.cc:407] Connection header received
0619 07:00:59.860354 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.860363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.860614 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.860780 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:59.861094 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.862347 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.864080 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.864946 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.865067 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.865218 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.865404 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.865502 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":167,"threads_started":1}
I20260619 07:00:59.865917 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.860396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.11.106.67:43955)
0619 07:00:59.860861 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.860874 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.860898 (+ 24us) server_negotiation.cc:407] Connection header received
0619 07:00:59.860968 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.860973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.861155 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.861565 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:59.861715 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.862698 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.864094 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.864946 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.865487 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.865621 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.865672 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.865768 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":94,"threads_started":1}
I20260619 07:00:59.866057 27576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.860602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:38839 (local address 127.0.0.1:46048)
0619 07:00:59.861528 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:59.861542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:59.861649 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:59.862036 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:59.862039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:59.862056 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:59.862169 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.862177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.863502 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.863506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.864496 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.864505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.865642 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.865645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.865746 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.865769 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 07:00:59.865845 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 07:00:59.865924 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260619 07:00:59.866505 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.861092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.11.106.65:38839)
0619 07:00:59.861513 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.861526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.861614 (+ 88us) server_negotiation.cc:407] Connection header received
0619 07:00:59.861758 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.861763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.861905 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.862060 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:59.862268 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.863364 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.864642 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.865358 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.865457 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.865584 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.866264 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.866374 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":115,"threads_started":1}
I20260619 07:00:59.866537 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.859483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:41001 (local address 127.0.0.1:35134)
0619 07:00:59.859958 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:59.860011 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:59.860149 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:59.860793 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:59.860799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:59.860822 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:59.860961 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.860968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.862490 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.862494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.863931 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.863943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.865072 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.865076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.865197 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.865241 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 07:00:59.865357 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 07:00:59.866413 (+ 1056us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":33,"thread_start_us":85,"threads_started":1}
I20260619 07:00:59.869278 27495 tablet_service.cc:1467] Processing CreateTablet for tablet a4aaac05bfbc4632a5f321b1d2a0676f (DEFAULT_TABLE table=test-table [id=ae4c4d24e13d4a74ab1d8a0ac474ea8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:59.869313 27361 tablet_service.cc:1467] Processing CreateTablet for tablet a4aaac05bfbc4632a5f321b1d2a0676f (DEFAULT_TABLE table=test-table [id=ae4c4d24e13d4a74ab1d8a0ac474ea8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:59.870273 27495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4aaac05bfbc4632a5f321b1d2a0676f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:59.870314 27361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4aaac05bfbc4632a5f321b1d2a0676f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:59.870352 27227 tablet_service.cc:1467] Processing CreateTablet for tablet a4aaac05bfbc4632a5f321b1d2a0676f (DEFAULT_TABLE table=test-table [id=ae4c4d24e13d4a74ab1d8a0ac474ea8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:00:59.871234 27227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4aaac05bfbc4632a5f321b1d2a0676f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:00:59.876384 27578 tablet_bootstrap.cc:492] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: Bootstrap starting.
I20260619 07:00:59.877972 27579 tablet_bootstrap.cc:492] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Bootstrap starting.
I20260619 07:00:59.878707 27578 tablet_bootstrap.cc:654] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:59.879858 27578 log.cc:824] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:59.880088 27579 tablet_bootstrap.cc:654] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:59.880837 27579 log.cc:824] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:59.880877 27580 tablet_bootstrap.cc:492] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Bootstrap starting.
I20260619 07:00:59.882194 27578 tablet_bootstrap.cc:492] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: No bootstrap required, opened a new log
I20260619 07:00:59.882391 27578 ts_tablet_manager.cc:1397] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 07:00:59.882568 27580 tablet_bootstrap.cc:654] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Neither blocks nor log segments found. Creating new log.
I20260619 07:00:59.883193 27580 log.cc:824] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Log is configured to *not* fsync() on all Append() calls
I20260619 07:00:59.884922 27580 tablet_bootstrap.cc:492] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: No bootstrap required, opened a new log
I20260619 07:00:59.885115 27580 ts_tablet_manager.cc:1397] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 07:00:59.885501 27578 raft_consensus.cc:348] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:00:59.885720 27578 raft_consensus.cc:374] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:59.885792 27578 raft_consensus.cc:729] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f2522d45ce41a9b29b5af9a802d0f9, State: Initialized, Role: FOLLOWER
I20260619 07:00:59.886216 27578 consensus_queue.cc:260] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:00:59.886879 27561 heartbeater.cc:503] Master 127.11.106.126:38655 was elected leader, sending a full tablet report...
I20260619 07:00:59.887162 27578 ts_tablet_manager.cc:1428] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:00:59.889006 27579 tablet_bootstrap.cc:492] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: No bootstrap required, opened a new log
I20260619 07:00:59.889250 27579 ts_tablet_manager.cc:1397] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260619 07:00:59.889243 27580 raft_consensus.cc:348] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:00:59.889508 27580 raft_consensus.cc:374] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:59.889611 27580 raft_consensus.cc:729] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72187e72c72a4c68897d702fb2e8c54b, State: Initialized, Role: FOLLOWER
I20260619 07:00:59.890064 27580 consensus_queue.cc:260] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:00:59.890699 27293 heartbeater.cc:503] Master 127.11.106.126:38655 was elected leader, sending a full tablet report...
I20260619 07:00:59.890969 27580 ts_tablet_manager.cc:1428] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 07:00:59.892942 27579 raft_consensus.cc:348] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:00:59.893221 27579 raft_consensus.cc:374] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:00:59.893278 27579 raft_consensus.cc:729] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a78f36fcfb4086a6aa5c0dd0a21279, State: Initialized, Role: FOLLOWER
I20260619 07:00:59.893687 27579 consensus_queue.cc:260] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:00:59.894332 27427 heartbeater.cc:503] Master 127.11.106.126:38655 was elected leader, sending a full tablet report...
I20260619 07:00:59.894662 27579 ts_tablet_manager.cc:1428] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:00:59.967374 27586 raft_consensus.cc:482] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:00:59.967662 27586 raft_consensus.cc:504] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:00:59.969249 27586 leader_election.cc:288] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955), 72187e72c72a4c68897d702fb2e8c54b (127.11.106.65:38839)
I20260619 07:00:59.973379 27429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.969573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43955 (local address 127.11.106.66:41229)
0619 07:00:59.969729 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:59.969744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:59.969854 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:59.970275 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:59.970279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:59.970297 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:59.970432 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.970440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.971288 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.971293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.972066 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.972074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.972860 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.972864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.973027 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.973052 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 07:00:59.973139 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 07:00:59.973211 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":25}
I20260619 07:00:59.973420 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.969661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:41229 (local address 127.11.106.67:43955)
0619 07:00:59.969783 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.969790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.969807 (+ 17us) server_negotiation.cc:407] Connection header received
0619 07:00:59.969975 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.969979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.970027 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.970102 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:59.970551 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.971174 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.972201 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.972746 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.972890 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.972963 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.973189 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.973275 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 07:00:59.974030 27515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4aaac05bfbc4632a5f321b1d2a0676f" candidate_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" is_pre_election: true
I20260619 07:00:59.974393 27515 raft_consensus.cc:2454] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a78f36fcfb4086a6aa5c0dd0a21279 in term 0.
I20260619 07:00:59.974635 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.969883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:38839 (local address 127.11.106.66:33449)
0619 07:00:59.970178 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 07:00:59.970192 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:00:59.970295 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:00:59.970584 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:00:59.970587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:00:59.970603 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:00:59.970774 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.970786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.971945 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.971951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.973251 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:00:59.973260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.974238 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.974241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:00:59.974329 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.974349 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 07:00:59.974423 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 07:00:59.974495 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":21,"thread_start_us":92,"threads_started":1}
I20260619 07:00:59.974697 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:00:59.970207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:33449 (local address 127.11.106.65:38839)
0619 07:00:59.970343 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 07:00:59.970349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:00:59.970363 (+ 14us) server_negotiation.cc:407] Connection header received
0619 07:00:59.970421 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:00:59.970424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:00:59.970502 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:00:59.970574 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:00:59.970931 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.971779 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.973383 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:00:59.973983 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:00:59.974081 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:00:59.974126 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 07:00:59.974482 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 07:00:59.974563 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 07:00:59.974949 27317 leader_election.cc:302] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f2522d45ce41a9b29b5af9a802d0f9, b4a78f36fcfb4086a6aa5c0dd0a21279; no voters:
I20260619 07:00:59.975241 27247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4aaac05bfbc4632a5f321b1d2a0676f" candidate_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72187e72c72a4c68897d702fb2e8c54b" is_pre_election: true
I20260619 07:00:59.975293 27586 raft_consensus.cc:2781] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:00:59.975374 27586 raft_consensus.cc:482] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:00:59.975426 27586 raft_consensus.cc:3037] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:59.975528 27247 raft_consensus.cc:2454] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a78f36fcfb4086a6aa5c0dd0a21279 in term 0.
I20260619 07:00:59.976673 27586 raft_consensus.cc:504] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:00:59.977053 27586 leader_election.cc:288] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [CANDIDATE]: Term 1 election: Requested vote from peers 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955), 72187e72c72a4c68897d702fb2e8c54b (127.11.106.65:38839)
I20260619 07:00:59.977451 27247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4aaac05bfbc4632a5f321b1d2a0676f" candidate_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72187e72c72a4c68897d702fb2e8c54b"
I20260619 07:00:59.977473 27515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4aaac05bfbc4632a5f321b1d2a0676f" candidate_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f2522d45ce41a9b29b5af9a802d0f9"
I20260619 07:00:59.977595 27247 raft_consensus.cc:3037] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:59.977602 27515 raft_consensus.cc:3037] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:00:59.978641 27515 raft_consensus.cc:2454] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a78f36fcfb4086a6aa5c0dd0a21279 in term 1.
I20260619 07:00:59.979008 27317 leader_election.cc:302] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f2522d45ce41a9b29b5af9a802d0f9, b4a78f36fcfb4086a6aa5c0dd0a21279; no voters:
I20260619 07:00:59.979041 27247 raft_consensus.cc:2454] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a78f36fcfb4086a6aa5c0dd0a21279 in term 1.
I20260619 07:00:59.979452 27586 raft_consensus.cc:2781] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:00:59.979701 27586 raft_consensus.cc:686] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 LEADER]: Becoming Leader. State: Replica: b4a78f36fcfb4086a6aa5c0dd0a21279, State: Running, Role: LEADER
I20260619 07:00:59.980047 27586 consensus_queue.cc:237] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:00:59.983457 27101 catalog_manager.cc:5495] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66). New cstate: current_term: 1 leader_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } health_report { overall_health: UNKNOWN } } }
I20260619 07:01:00.017894 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:00.014487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.11.106.66:41001)
0619 07:01:00.014622 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:00.014626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:00.014651 (+ 25us) server_negotiation.cc:407] Connection header received
0619 07:01:00.014769 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:00.014771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:00.014826 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:00.014916 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:01:00.015120 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:00.015606 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:00.016415 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:00.016702 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:00.016784 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:00.017072 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:01:00.017503 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:01:00.017595 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:00.017660 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 07:01:00.017713 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 07:01:00.024207 27247 raft_consensus.cc:1261] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Refusing update from remote peer b4a78f36fcfb4086a6aa5c0dd0a21279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:01:00.024204 27515 raft_consensus.cc:1261] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [term 1 FOLLOWER]: Refusing update from remote peer b4a78f36fcfb4086a6aa5c0dd0a21279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:01:00.025002 27588 consensus_queue.cc:1035] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:01:00.025224 27586 consensus_queue.cc:1035] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:01:00.034868 27593 mvcc.cc:205] Tried to move back new op lower bound from 7298467676251140096 to 7298467676081655808. Current Snapshot: MvccSnapshot[applied={T|T < 7298467676251140096}]
I20260619 07:01:00.041329 27102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:01:00.041486 27102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:01:00.043815 27102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Sending DeleteTablet for 3 replicas of tablet a4aaac05bfbc4632a5f321b1d2a0676f
I20260619 07:01:00.044602 27361 tablet_service.cc:1514] Processing DeleteTablet for tablet a4aaac05bfbc4632a5f321b1d2a0676f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:00 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:35134
I20260619 07:01:00.044705 27495 tablet_service.cc:1514] Processing DeleteTablet for tablet a4aaac05bfbc4632a5f321b1d2a0676f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:00 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:44966
I20260619 07:01:00.044826 27227 tablet_service.cc:1514] Processing DeleteTablet for tablet a4aaac05bfbc4632a5f321b1d2a0676f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:00 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:46048
I20260619 07:01:00.045280 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 27164
I20260619 07:01:00.045339 27603 tablet_replica.cc:330] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: stopping tablet replica
I20260619 07:01:00.045555 27601 tablet_replica.cc:330] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: stopping tablet replica
I20260619 07:01:00.045691 27603 raft_consensus.cc:2229] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:01:00.046008 27601 raft_consensus.cc:2229] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:01:00.046355 27601 raft_consensus.cc:2258] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:01:00.046732 27602 tablet_replica.cc:330] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: stopping tablet replica
I20260619 07:01:00.047008 27602 raft_consensus.cc:2229] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 LEADER]: Raft consensus shutting down.
I20260619 07:01:00.047423 27602 raft_consensus.cc:2258] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:01:00.049129 27601 ts_tablet_manager.cc:1905] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:01:00.051905 27601 ts_tablet_manager.cc:1918] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:01:00.052007 27601 log.cc:1196] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/a4aaac05bfbc4632a5f321b1d2a0676f
I20260619 07:01:00.052352 27601 ts_tablet_manager.cc:1939] T a4aaac05bfbc4632a5f321b1d2a0676f P 36f2522d45ce41a9b29b5af9a802d0f9: Deleting consensus metadata
I20260619 07:01:00.053123 27602 ts_tablet_manager.cc:1905] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:01:00.053674 27089 catalog_manager.cc:4841] TS 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955): tablet a4aaac05bfbc4632a5f321b1d2a0676f (table test-table [id=ae4c4d24e13d4a74ab1d8a0ac474ea8c]) successfully deleted
W20260619 07:01:00.054319 27089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:38839 (error 108)
I20260619 07:01:00.054575 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 27298
I20260619 07:01:00.056432 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:00.055828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:38839 (local address 127.0.0.1:46054)
0619 07:01:00.056242 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:00.056327 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
W20260619 07:01:00.056619 27089 catalog_manager.cc:4584] TS 72187e72c72a4c68897d702fb2e8c54b (127.11.106.65:38839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4aaac05bfbc4632a5f321b1d2a0676f: Network error: Client connection negotiation failed: client connection to 127.11.106.65:38839: connect: Connection refused (error 111)
I20260619 07:01:00.061444 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 27432
I20260619 07:01:00.061672 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:00.061361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:41001 (local address 127.0.0.1:35150)
0619 07:01:00.061491 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:00.061576 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.66:41001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 07:01:00.067029 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 27070
I20260619 07:01:20.073554 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38655
--webserver_interface=127.11.106.126
--webserver_port=41631
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:38655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:01:20.193146 27605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:20.193408 27605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:20.193449 27605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:20.197364 27605 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 07:01:20.197409 27605 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 07:01:20.197425 27605 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 07:01:20.197439 27605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:01:20.197458 27605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:20.197475 27605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:20.197491 27605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:01:20.197510 27605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:01:20.202730 27605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:38655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=41631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:20.203712 27605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:20.204697 27605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:20.220459 27614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:20.220455 27611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:20.220499 27612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:20.220808 27605 server_base.cc:1034] running on GCE node
I20260619 07:01:20.221474 27605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:20.222669 27605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:20.223879 27605 hybrid_clock.cc:648] HybridClock initialized: now 1781852480223865 us; error 36 us; skew 500 ppm
Jun 19 07:01:20 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:20.226725 27605 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:01:20.227591 27605 webserver.cc:460] Webserver started at http://127.11.106.126:41631/ using document root <none> and password file <none>
I20260619 07:01:20.228142 27605 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:20.228210 27605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:20.232100 27605 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:01:20.234365 27621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:20.235708 27605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:01:20.235855 27605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "93c478b22a0342a28a7152e2206fabc0"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "24eaee16d069da5a8985551b97854d74"
server_key_iv: "cbb7b53b3defcffef1aaa201a0eec871"
server_key_version: "encryptionkey@0"
I20260619 07:01:20.236583 27605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:20.248961 27605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:20.251969 27605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:20.252203 27605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:20.259770 27605 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:38655
I20260619 07:01:20.260093 27673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:38655 every 8 connection(s)
I20260619 07:01:20.260994 27605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:01:20.261456 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 27605
I20260619 07:01:20.261968 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:38839
--local_ip_for_outbound_sockets=127.11.106.65
--tserver_master_addrs=127.11.106.126:38655
--webserver_port=39103
--webserver_interface=127.11.106.65
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:01:20.266191 27674 sys_catalog.cc:263] Verifying existing consensus state
I20260619 07:01:20.267904 27674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Bootstrap starting.
I20260619 07:01:20.285231 27674 log.cc:824] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Log is configured to *not* fsync() on all Append() calls
I20260619 07:01:20.293857 27674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 07:01:20.294245 27674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Bootstrap complete.
I20260619 07:01:20.298422 27674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:01:20.298818 27674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c478b22a0342a28a7152e2206fabc0, State: Initialized, Role: FOLLOWER
I20260619 07:01:20.299304 27674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:01:20.299414 27674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:01:20.299505 27674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:01:20.299630 27674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 1 FOLLOWER]: Advancing to term 2
I20260619 07:01:20.301553 27674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:01:20.301894 27674 leader_election.cc:302] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93c478b22a0342a28a7152e2206fabc0; no voters:
I20260619 07:01:20.302239 27674 leader_election.cc:288] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 07:01:20.302327 27678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 07:01:20.302542 27678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 2 LEADER]: Becoming Leader. State: Replica: 93c478b22a0342a28a7152e2206fabc0, State: Running, Role: LEADER
I20260619 07:01:20.302834 27678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:01:20.303481 27674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:01:20.303829 27678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93c478b22a0342a28a7152e2206fabc0. Latest consensus state: current_term: 2 leader_uuid: "93c478b22a0342a28a7152e2206fabc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } } }
I20260619 07:01:20.303948 27678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: This master's current role is: LEADER
I20260619 07:01:20.304327 27682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:01:20.304368 27679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "93c478b22a0342a28a7152e2206fabc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } } }
I20260619 07:01:20.304642 27679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: This master's current role is: LEADER
I20260619 07:01:20.309779 27682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:01:20.310596 27682 catalog_manager.cc:1259] Loaded cluster ID: bdbef68fcc5845d08e755308da02fd41
I20260619 07:01:20.310686 27682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:01:20.312443 27682 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:01:20.313083 27682 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Loaded TSK: 0
I20260619 07:01:20.323724 27682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Generated new TSK 1
I20260619 07:01:20.324023 27682 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 07:01:20.389364 27676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:20.389660 27676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:20.389751 27676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:20.393774 27676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:20.393854 27676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:20.393942 27676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:01:20.399210 27676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:38839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=39103
--tserver_master_addrs=127.11.106.126:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:20.400274 27676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:20.401372 27676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:20.411501 27704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:20.411463 27702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:20.411589 27701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:20.411758 27676 server_base.cc:1034] running on GCE node
I20260619 07:01:20.412263 27676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:20.413038 27676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:20.414223 27676 hybrid_clock.cc:648] HybridClock initialized: now 1781852480414188 us; error 48 us; skew 500 ppm
Jun 19 07:01:20 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:20.417027 27676 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:01:20.417985 27676 webserver.cc:460] Webserver started at http://127.11.106.65:39103/ using document root <none> and password file <none>
I20260619 07:01:20.418637 27676 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:20.418711 27676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:20.422749 27676 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:01:20.424988 27711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:20.426342 27676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:01:20.426435 27676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "72187e72c72a4c68897d702fb2e8c54b"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "b6d13ce2de93b182ed3152a05db721bf"
server_key_iv: "c5f5fca4e50432546a877b943c219373"
server_key_version: "encryptionkey@0"
I20260619 07:01:20.427003 27676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:20.442456 27676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:20.445809 27676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:20.446015 27676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:20.446676 27676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:20.447824 27718 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 07:01:20.451961 27676 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 07:01:20.452025 27676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 07:01:20.452072 27676 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 07:01:20.453887 27676 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 07:01:20.453943 27676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 07:01:20.454214 27718 tablet_bootstrap.cc:492] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Bootstrap starting.
I20260619 07:01:20.464882 27676 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:38839
I20260619 07:01:20.465075 27825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:38839 every 8 connection(s)
I20260619 07:01:20.466090 27676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:01:20.469316 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 27676
I20260619 07:01:20.469857 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:41001
--local_ip_for_outbound_sockets=127.11.106.66
--tserver_master_addrs=127.11.106.126:38655
--webserver_port=43571
--webserver_interface=127.11.106.66
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:01:20.475479 27718 log.cc:824] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Log is configured to *not* fsync() on all Append() calls
Jun 19 07:01:20 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:01:20.484288 27718 tablet_bootstrap.cc:492] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 07:01:20.484762 27718 tablet_bootstrap.cc:492] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Bootstrap complete.
I20260619 07:01:20.486016 27718 ts_tablet_manager.cc:1397] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20260619 07:01:20.486824 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.468321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:39493 (local address 127.11.106.126:38655)
0619 07:01:20.468698 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:20.468724 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:20.469201 (+ 477us) server_negotiation.cc:407] Connection header received
0619 07:01:20.471034 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:20.471050 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:20.471478 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:20.471947 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:20.473239 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.474724 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.476050 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.476858 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.477046 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.480258 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:20.480265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:20.480280 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:20.480312 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:20.482846 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.483581 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.483588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.483594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.483743 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.484205 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.484208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.484210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.484726 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:20.484873 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:20.485638 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 07:01:20.485963 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":132,"threads_started":1}
I20260619 07:01:20.486850 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.468327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:39493)
0619 07:01:20.468977 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:20.469077 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:20.470543 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:20.472016 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:20.472028 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:20.472576 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:20.473050 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.473064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.474885 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.474888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.475845 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.475855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.477022 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.477027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.477216 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.477956 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:01:20.477984 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:20.480039 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:20.482991 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:20.482999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:20.483018 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:20.483442 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:20.483862 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:20.483867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:20.483870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:20.484074 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:20.484911 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:20.484921 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:20.485278 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 07:01:20.485499 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 07:01:20.485833 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":210,"threads_started":1}
I20260619 07:01:20.489048 27718 raft_consensus.cc:348] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:20.489586 27718 raft_consensus.cc:729] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72187e72c72a4c68897d702fb2e8c54b, State: Initialized, Role: FOLLOWER
I20260619 07:01:20.489679 27826 heartbeater.cc:347] Connected to a master server at 127.11.106.126:38655
I20260619 07:01:20.489912 27826 heartbeater.cc:464] Registering TS with master...
I20260619 07:01:20.490125 27718 consensus_queue.cc:260] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:20.490446 27826 heartbeater.cc:511] Master 127.11.106.126:38655 requested a full tablet report, sending...
I20260619 07:01:20.491062 27718 ts_tablet_manager.cc:1428] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:01:20.493278 27638 ts_manager.cc:194] Registered new tserver with Master: 72187e72c72a4c68897d702fb2e8c54b (127.11.106.65:38839)
I20260619 07:01:20.497589 27638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:39493
I20260619 07:01:20.500408 27826 heartbeater.cc:503] Master 127.11.106.126:38655 was elected leader, sending a full tablet report...
Jun 19 07:01:20 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.65@KRBTEST.COM
I20260619 07:01:20.507670 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.496594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:38839 (local address 127.0.0.1:55290)
0619 07:01:20.497023 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:20.497067 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:20.497190 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:20.497866 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:20.497873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:20.498241 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:20.498344 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.498351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.499772 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.499775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.500712 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.500724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.501440 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.501444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.501551 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.502203 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:01:20.502215 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:20.503733 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:20.505875 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:20.505886 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:20.505891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:20.506128 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:20.506437 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:20.506440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:20.506442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:20.506531 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:20.507008 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:20.507014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:20.507133 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 07:01:20.507274 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 07:01:20.507475 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":128,"threads_started":1}
I20260619 07:01:20.507692 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.496629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.11.106.65:38839)
0619 07:01:20.497370 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:20.497380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:20.497397 (+ 17us) server_negotiation.cc:407] Connection header received
0619 07:01:20.497451 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:20.497455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:20.497658 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:20.497814 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:20.498518 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.499623 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.500843 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.501316 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.501438 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.503875 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:20.503887 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:20.503903 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:20.503924 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:20.505769 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.506247 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.506254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.506255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.506342 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.506633 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.506636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.506638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.506875 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:20.507012 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:20.507368 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 07:01:20.507542 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":642,"thread_start_us":97,"threads_started":1}
I20260619 07:01:20.509037 27758 tablet_service.cc:1514] Processing DeleteTablet for tablet a4aaac05bfbc4632a5f321b1d2a0676f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:00 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:55290
I20260619 07:01:20.509896 27835 tablet_replica.cc:330] stopping tablet replica
I20260619 07:01:20.510224 27835 raft_consensus.cc:2229] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:01:20.510480 27835 raft_consensus.cc:2258] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:01:20.511613 27835 ts_tablet_manager.cc:1905] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:01:20.516273 27835 ts_tablet_manager.cc:1918] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:01:20.516376 27835 log.cc:1196] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal/wals/a4aaac05bfbc4632a5f321b1d2a0676f
I20260619 07:01:20.516665 27835 ts_tablet_manager.cc:1939] T a4aaac05bfbc4632a5f321b1d2a0676f P 72187e72c72a4c68897d702fb2e8c54b: Deleting consensus metadata
I20260619 07:01:20.518915 27625 catalog_manager.cc:4841] TS 72187e72c72a4c68897d702fb2e8c54b (127.11.106.65:38839): tablet a4aaac05bfbc4632a5f321b1d2a0676f (table test-table [id=ae4c4d24e13d4a74ab1d8a0ac474ea8c]) successfully deleted
W20260619 07:01:20.596691 27830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:20.596964 27830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:20.597009 27830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:20.601085 27830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:20.601141 27830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:20.601200 27830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:01:20.606771 27830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:41001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=43571
--tserver_master_addrs=127.11.106.126:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:20.607977 27830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:20.609050 27830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:20.618633 27841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:20.618631 27840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:20.618633 27843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:20.618896 27830 server_base.cc:1034] running on GCE node
I20260619 07:01:20.619395 27830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:20.620115 27830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:20.621300 27830 hybrid_clock.cc:648] HybridClock initialized: now 1781852480621280 us; error 35 us; skew 500 ppm
Jun 19 07:01:20 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:20.624128 27830 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:01:20.625092 27830 webserver.cc:460] Webserver started at http://127.11.106.66:43571/ using document root <none> and password file <none>
I20260619 07:01:20.625734 27830 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:20.625854 27830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:20.629817 27830 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:01:20.632165 27850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:20.633510 27830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:01:20.633644 27830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "87b0b40567eb76d085a220d6b485ce83"
server_key_iv: "ee74407e502732390ba32d708dd2e90e"
server_key_version: "encryptionkey@0"
I20260619 07:01:20.634186 27830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:20.673607 27830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:20.677307 27830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:20.677523 27830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:20.678186 27830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:20.679276 27857 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 07:01:20.683207 27857 ts_tablet_manager.cc:1852] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 07:01:20.683590 27857 ts_tablet_manager.cc:1905] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:01:20.687572 27857 ts_tablet_manager.cc:1918] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:01:20.687681 27857 log.cc:1196] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/a4aaac05bfbc4632a5f321b1d2a0676f
I20260619 07:01:20.688014 27857 ts_tablet_manager.cc:1939] T a4aaac05bfbc4632a5f321b1d2a0676f P b4a78f36fcfb4086a6aa5c0dd0a21279: Deleting consensus metadata
I20260619 07:01:20.688338 27830 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 07:01:20.688431 27830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260619 07:01:20.688514 27830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:01:20.688568 27830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:20.697642 27830 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:41001
I20260619 07:01:20.697682 27964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:41001 every 8 connection(s)
I20260619 07:01:20.698511 27830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:01:20.701426 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 27830
I20260619 07:01:20.701872 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:43955
--local_ip_for_outbound_sockets=127.11.106.67
--tserver_master_addrs=127.11.106.126:38655
--webserver_port=42227
--webserver_interface=127.11.106.67
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 07:01:20 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:01:20.713973 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.700305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:37745 (local address 127.11.106.126:38655)
0619 07:01:20.700443 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:20.700447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:20.701351 (+ 904us) server_negotiation.cc:407] Connection header received
0619 07:01:20.702406 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:20.702409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:20.702469 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:20.702582 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:20.704106 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.704812 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.706624 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.706848 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.706933 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.710082 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:20.710085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:20.710087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:20.710112 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:20.711589 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.712283 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.712285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.712286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.712359 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.712797 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.712799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.712800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.712974 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:20.713054 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:20.713735 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 07:01:20.713845 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 07:01:20.715044 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.700747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:37745)
0619 07:01:20.701203 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:20.701245 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:20.702197 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:20.702747 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:20.702754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:20.703221 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:20.703950 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.703973 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.704941 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.704947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.706488 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.706504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.707018 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.707025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.707266 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.707930 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:01:20.707956 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:20.709884 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:20.711742 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:20.711753 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:20.711770 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:20.712178 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:20.712455 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:20.712461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:20.712464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:20.712698 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:20.713087 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:20.713097 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:20.713458 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 07:01:20.713684 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 07:01:20.714030 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260619 07:01:20.716347 27965 heartbeater.cc:347] Connected to a master server at 127.11.106.126:38655
I20260619 07:01:20.716671 27965 heartbeater.cc:464] Registering TS with master...
I20260619 07:01:20.717224 27965 heartbeater.cc:511] Master 127.11.106.126:38655 requested a full tablet report, sending...
I20260619 07:01:20.718312 27638 ts_manager.cc:194] Registered new tserver with Master: b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001)
I20260619 07:01:20.718887 27638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:37745
W20260619 07:01:20.817163 27968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:20.817440 27968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:20.817548 27968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:20.822196 27968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:20.822280 27968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:20.822368 27968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:01:20.827615 27968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:43955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=42227
--tserver_master_addrs=127.11.106.126:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:20.828796 27968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:20.829830 27968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:20.839653 27974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:20.839747 27973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:20.839661 27976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:20.839962 27968 server_base.cc:1034] running on GCE node
I20260619 07:01:20.840335 27968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:20.841074 27968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:20.842260 27968 hybrid_clock.cc:648] HybridClock initialized: now 1781852480842227 us; error 46 us; skew 500 ppm
Jun 19 07:01:20 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:20.844862 27968 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:01:20.845682 27968 webserver.cc:460] Webserver started at http://127.11.106.67:42227/ using document root <none> and password file <none>
I20260619 07:01:20.846244 27968 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:20.846354 27968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:20.850118 27968 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 07:01:20.852175 27983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:20.853425 27968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:01:20.853564 27968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "36f2522d45ce41a9b29b5af9a802d0f9"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "7c5136418f4027a9e20750e8b8e407bc"
server_key_iv: "e3aaad4d06cae2dcffed9437566ad048"
server_key_version: "encryptionkey@0"
I20260619 07:01:20.854117 27968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:20.883912 27968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:20.886875 27968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:20.887087 27968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:20.887780 27968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:20.888636 27968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:01:20.888707 27968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:20.888777 27968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:01:20.888820 27968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:20.897770 27968 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:43955
I20260619 07:01:20.897800 28096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:43955 every 8 connection(s)
I20260619 07:01:20.898780 27968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:01:20.899034 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 27968
Jun 19 07:01:20 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852480, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:01:20.909518 27828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.899860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.11.106.126:38655)
0619 07:01:20.900020 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:20.900025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:20.900640 (+ 615us) server_negotiation.cc:407] Connection header received
0619 07:01:20.900688 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:20.900690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:20.900732 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:20.900826 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:20.901713 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.902339 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.903531 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.903741 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.903820 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.905351 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:20.905356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:20.905359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:20.905386 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:20.907121 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.907569 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.907572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.907573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.907700 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.908109 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.908112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.908113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.908270 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:20.908420 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:20.909221 (+ 801us) server_negotiation.cc:299] Negotiation successful
0619 07:01:20.909332 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 07:01:20.913053 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.900674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:38297 (local address 127.11.106.126:38655)
0619 07:01:20.900940 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:20.900945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:20.902026 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 07:01:20.903099 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:20.903102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:20.903142 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:20.903209 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:20.904776 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.905352 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.906601 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.906981 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.907106 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.909740 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:20.909743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:20.909744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:20.909764 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:20.911203 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.911743 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.911746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.911747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.911822 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.912092 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.912094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.912095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.912243 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:20.912314 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:20.912846 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 07:01:20.912935 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260619 07:01:20.913873 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.901175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:38297)
0619 07:01:20.901890 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:20.901930 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:20.902877 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:20.903416 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:20.903425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:20.903852 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:20.904619 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.904643 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.905500 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.905504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.906469 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.906479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.907131 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.907134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.907345 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.907928 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:01:20.907953 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:20.909554 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:20.911309 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:20.911313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:20.911325 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:20.911643 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:20.911881 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:20.911883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:20.911884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:20.912003 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:20.912329 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:20.912334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:20.912615 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 07:01:20.912784 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 07:01:20.913021 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":107,"threads_started":1}
I20260619 07:01:20.914970 28097 heartbeater.cc:347] Connected to a master server at 127.11.106.126:38655
I20260619 07:01:20.915185 28097 heartbeater.cc:464] Registering TS with master...
I20260619 07:01:20.915611 28097 heartbeater.cc:511] Master 127.11.106.126:38655 requested a full tablet report, sending...
I20260619 07:01:20.916517 27638 ts_manager.cc:194] Registered new tserver with Master: 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955)
I20260619 07:01:20.917117 27638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:38297
I20260619 07:01:20.922963 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:01:20.926789 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.923595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.11.106.126:38655)
0619 07:01:20.923732 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:20.923735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:20.923910 (+ 175us) server_negotiation.cc:407] Connection header received
0619 07:01:20.924037 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:20.924039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:20.924131 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:20.924216 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:01:20.924426 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.924972 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.925810 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.926025 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.926104 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.926370 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:01:20.926497 (+ 127us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 07:01:20.926696 (+ 199us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51586: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 07:01:20.926873 28100 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51586: token expired
W20260619 07:01:20.927198 27567 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.106.126:38655)
I20260619 07:01:20.934103 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.927559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.11.106.126:38655)
0619 07:01:20.927682 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:20.927685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:20.927697 (+ 12us) server_negotiation.cc:407] Connection header received
0619 07:01:20.927776 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:20.927781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:20.927825 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:20.927901 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:20.928453 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.928987 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.929704 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.929921 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.929995 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.931409 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:20.931412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:20.931414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:20.931434 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:20.932799 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.933175 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.933178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.933179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.933247 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:20.933466 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:20.933468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:20.933469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:20.933630 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:20.933725 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:20.933915 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 07:01:20.934000 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 07:01:20.936249 27638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:01:20.939560 27638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:01:20.947427 27758 tablet_service.cc:1467] Processing CreateTablet for tablet 02b25e789f5145ce993abd2df8954286 (DEFAULT_TABLE table=test-table [id=fb9155c09af746869cc2b0cf47e4cbaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:01:20.948077 27758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b25e789f5145ce993abd2df8954286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:01:20.950707 27718 tablet_bootstrap.cc:492] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Bootstrap starting.
I20260619 07:01:20.951475 27718 tablet_bootstrap.cc:654] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Neither blocks nor log segments found. Creating new log.
I20260619 07:01:20.952227 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.946294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43955 (local address 127.0.0.1:45558)
0619 07:01:20.946396 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:20.946411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:20.946487 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:20.947327 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:20.947331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:20.947350 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:01:20.947449 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.947459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.949400 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.949404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.950577 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.950597 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.951691 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.951695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.951818 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.951881 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 07:01:20.951983 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 07:01:20.952065 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":3}
I20260619 07:01:20.952293 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.946455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.11.106.67:43955)
0619 07:01:20.946787 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:20.946797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:20.946838 (+ 41us) server_negotiation.cc:407] Connection header received
0619 07:01:20.946921 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:20.946926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:20.947154 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:20.947624 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:01:20.947790 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.949259 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.950737 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.951570 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.951685 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.951827 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:20.952023 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 07:01:20.952138 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":136,"threads_started":1}
I20260619 07:01:20.952391 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.946526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:41001 (local address 127.0.0.1:53562)
0619 07:01:20.947115 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:20.947129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:20.947227 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:20.947873 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:20.947876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:20.947893 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:01:20.947981 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.947987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.949245 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.949248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.950613 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:20.950621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.951906 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.951910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:20.952039 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.952062 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 07:01:20.952142 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 07:01:20.952209 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":82,"thread_start_us":89,"threads_started":1}
I20260619 07:01:20.953100 27718 tablet_bootstrap.cc:492] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: No bootstrap required, opened a new log
I20260619 07:01:20.953192 27718 ts_tablet_manager.cc:1397] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 07:01:20.953227 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:20.947059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.11.106.66:41001)
0619 07:01:20.947451 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:20.947461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:20.947478 (+ 17us) server_negotiation.cc:407] Connection header received
0619 07:01:20.947538 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:20.947541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:20.947737 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:20.947880 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:01:20.948106 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.949126 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.950737 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:20.951570 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:20.951685 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:20.951827 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:20.952812 (+ 985us) server_negotiation.cc:299] Negotiation successful
0619 07:01:20.952912 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":112,"threads_started":1}
I20260619 07:01:20.953616 27718 raft_consensus.cc:348] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:20.953738 27718 raft_consensus.cc:374] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:01:20.953800 27718 raft_consensus.cc:729] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72187e72c72a4c68897d702fb2e8c54b, State: Initialized, Role: FOLLOWER
I20260619 07:01:20.953943 27718 consensus_queue.cc:260] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:20.954201 27718 ts_tablet_manager.cc:1428] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 07:01:20.954648 27899 tablet_service.cc:1467] Processing CreateTablet for tablet 02b25e789f5145ce993abd2df8954286 (DEFAULT_TABLE table=test-table [id=fb9155c09af746869cc2b0cf47e4cbaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:01:20.955510 27899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b25e789f5145ce993abd2df8954286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:01:20.955717 28031 tablet_service.cc:1467] Processing CreateTablet for tablet 02b25e789f5145ce993abd2df8954286 (DEFAULT_TABLE table=test-table [id=fb9155c09af746869cc2b0cf47e4cbaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:01:20.956758 28031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b25e789f5145ce993abd2df8954286. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:01:20.959964 27857 tablet_bootstrap.cc:492] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: Bootstrap starting.
I20260619 07:01:20.961885 27857 tablet_bootstrap.cc:654] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: Neither blocks nor log segments found. Creating new log.
I20260619 07:01:20.962587 27857 log.cc:824] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: Log is configured to *not* fsync() on all Append() calls
I20260619 07:01:20.963261 28111 tablet_bootstrap.cc:492] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: Bootstrap starting.
I20260619 07:01:20.964514 27857 tablet_bootstrap.cc:492] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: No bootstrap required, opened a new log
I20260619 07:01:20.964737 27857 ts_tablet_manager.cc:1397] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 07:01:20.965575 28111 tablet_bootstrap.cc:654] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: Neither blocks nor log segments found. Creating new log.
I20260619 07:01:20.966236 28111 log.cc:824] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: Log is configured to *not* fsync() on all Append() calls
W20260619 07:01:20.967296 27827 tablet.cc:2369] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:01:20.968255 28111 tablet_bootstrap.cc:492] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: No bootstrap required, opened a new log
I20260619 07:01:20.968096 27857 raft_consensus.cc:348] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:20.968353 27857 raft_consensus.cc:374] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:01:20.968422 27857 raft_consensus.cc:729] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a78f36fcfb4086a6aa5c0dd0a21279, State: Initialized, Role: FOLLOWER
I20260619 07:01:20.968459 28111 ts_tablet_manager.cc:1397] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 07:01:20.968854 27857 consensus_queue.cc:260] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:20.969472 27965 heartbeater.cc:503] Master 127.11.106.126:38655 was elected leader, sending a full tablet report...
I20260619 07:01:20.969632 27857 ts_tablet_manager.cc:1428] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:01:20.972005 28111 raft_consensus.cc:348] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:20.972270 28111 raft_consensus.cc:374] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:01:20.972328 28111 raft_consensus.cc:729] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f2522d45ce41a9b29b5af9a802d0f9, State: Initialized, Role: FOLLOWER
I20260619 07:01:20.972779 28111 consensus_queue.cc:260] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:20.973385 28097 heartbeater.cc:503] Master 127.11.106.126:38655 was elected leader, sending a full tablet report...
I20260619 07:01:20.973724 28111 ts_tablet_manager.cc:1428] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:01:21.012334 28113 raft_consensus.cc:482] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:01:21.012629 28113 raft_consensus.cc:504] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:21.013855 28113 leader_election.cc:288] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955), 72187e72c72a4c68897d702fb2e8c54b (127.11.106.65:38839)
I20260619 07:01:21.017724 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:21.014113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43955 (local address 127.11.106.66:44881)
0619 07:01:21.014289 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:21.014308 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:21.014414 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:21.014724 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:21.014727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:21.014744 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:01:21.014882 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:21.014888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.015743 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.015750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:21.016635 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:21.016653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.017284 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.017287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:21.017392 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:21.017414 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 07:01:21.017514 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 07:01:21.017585 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":1}
I20260619 07:01:21.017724 28106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:21.014202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:44881 (local address 127.11.106.67:43955)
0619 07:01:21.014347 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:21.014354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:21.014372 (+ 18us) server_negotiation.cc:407] Connection header received
0619 07:01:21.014561 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:21.014567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:21.014630 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:21.014716 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:01:21.014998 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.015633 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.016763 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.017162 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.017294 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:21.017350 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:21.017529 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 07:01:21.017599 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 07:01:21.018443 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:21.014417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:38839 (local address 127.11.106.66:46151)
0619 07:01:21.014781 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:21.014796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:21.014887 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:21.015193 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:21.015196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:21.015243 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:01:21.015364 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:21.015368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.016111 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.016117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:21.017032 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:21.017042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.017714 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.017718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:21.018125 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:21.018151 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 07:01:21.018248 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 07:01:21.018312 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":116,"threads_started":1}
I20260619 07:01:21.018455 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:21.014552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:46151 (local address 127.11.106.65:38839)
0619 07:01:21.014848 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:21.014853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:21.014867 (+ 14us) server_negotiation.cc:407] Connection header received
0619 07:01:21.014981 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:21.014984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:21.015059 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:21.015173 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:01:21.015486 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.015989 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.017156 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.017594 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.017726 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:21.017903 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:21.018277 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 07:01:21.018344 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":91,"threads_started":1}
I20260619 07:01:21.018430 28051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b25e789f5145ce993abd2df8954286" candidate_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" is_pre_election: true
I20260619 07:01:21.018919 28051 raft_consensus.cc:2454] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a78f36fcfb4086a6aa5c0dd0a21279 in term 0.
I20260619 07:01:21.019376 27778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b25e789f5145ce993abd2df8954286" candidate_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72187e72c72a4c68897d702fb2e8c54b" is_pre_election: true
I20260619 07:01:21.019464 27854 leader_election.cc:302] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f2522d45ce41a9b29b5af9a802d0f9, b4a78f36fcfb4086a6aa5c0dd0a21279; no voters:
I20260619 07:01:21.019701 27778 raft_consensus.cc:2454] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4a78f36fcfb4086a6aa5c0dd0a21279 in term 0.
I20260619 07:01:21.019804 28113 raft_consensus.cc:2781] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:01:21.019901 28113 raft_consensus.cc:482] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:01:21.019958 28113 raft_consensus.cc:3037] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:01:21.021077 28113 raft_consensus.cc:504] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:21.021425 28113 leader_election.cc:288] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [CANDIDATE]: Term 1 election: Requested vote from peers 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955), 72187e72c72a4c68897d702fb2e8c54b (127.11.106.65:38839)
I20260619 07:01:21.021853 28051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b25e789f5145ce993abd2df8954286" candidate_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f2522d45ce41a9b29b5af9a802d0f9"
I20260619 07:01:21.021955 27778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b25e789f5145ce993abd2df8954286" candidate_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72187e72c72a4c68897d702fb2e8c54b"
I20260619 07:01:21.021960 28051 raft_consensus.cc:3037] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:01:21.022094 27778 raft_consensus.cc:3037] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:01:21.023053 28051 raft_consensus.cc:2454] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a78f36fcfb4086a6aa5c0dd0a21279 in term 1.
I20260619 07:01:21.023315 27778 raft_consensus.cc:2454] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4a78f36fcfb4086a6aa5c0dd0a21279 in term 1.
I20260619 07:01:21.023379 27854 leader_election.cc:302] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f2522d45ce41a9b29b5af9a802d0f9, b4a78f36fcfb4086a6aa5c0dd0a21279; no voters:
I20260619 07:01:21.023535 28113 raft_consensus.cc:2781] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:01:21.023777 28113 raft_consensus.cc:686] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 LEADER]: Becoming Leader. State: Replica: b4a78f36fcfb4086a6aa5c0dd0a21279, State: Running, Role: LEADER
I20260619 07:01:21.024145 28113 consensus_queue.cc:237] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:21.027669 27638 catalog_manager.cc:5495] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66). New cstate: current_term: 1 leader_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } health_report { overall_health: UNKNOWN } } }
I20260619 07:01:21.055305 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:21.051618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.11.106.66:41001)
0619 07:01:21.051753 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:21.051757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:21.051770 (+ 13us) server_negotiation.cc:407] Connection header received
0619 07:01:21.051869 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:21.051871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:21.051925 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:21.052014 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:01:21.052230 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.052752 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.053583 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:21.053792 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:21.053874 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:21.054185 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:01:21.054691 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:01:21.054795 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:21.055020 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 07:01:21.055110 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 07:01:21.061795 27778 raft_consensus.cc:1261] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Refusing update from remote peer b4a78f36fcfb4086a6aa5c0dd0a21279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:01:21.061822 28051 raft_consensus.cc:1261] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [term 1 FOLLOWER]: Refusing update from remote peer b4a78f36fcfb4086a6aa5c0dd0a21279: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:01:21.062595 28117 consensus_queue.cc:1035] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:01:21.062814 28113 consensus_queue.cc:1035] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:01:21.067746 28119 mvcc.cc:205] Tried to move back new op lower bound from 7298467762421059584 to 7298467762277986304. Current Snapshot: MvccSnapshot[applied={T|T < 7298467762421059584}]
I20260619 07:01:21.071261 28122 mvcc.cc:205] Tried to move back new op lower bound from 7298467762421059584 to 7298467762277986304. Current Snapshot: MvccSnapshot[applied={T|T < 7298467762421059584}]
I20260619 07:01:21.078442 27637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:01:21.078621 27637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:01:21.081079 27637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Sending DeleteTablet for 3 replicas of tablet 02b25e789f5145ce993abd2df8954286
I20260619 07:01:21.081761 28031 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b25e789f5145ce993abd2df8954286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:21 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:45558
I20260619 07:01:21.081918 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 27676
I20260619 07:01:21.081945 27758 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b25e789f5145ce993abd2df8954286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:21 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:55290
I20260619 07:01:21.081930 27899 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b25e789f5145ce993abd2df8954286 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:21 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:53562
I20260619 07:01:21.085148 28130 tablet_replica.cc:330] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: stopping tablet replica
I20260619 07:01:21.085489 28130 raft_consensus.cc:2229] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 LEADER]: Raft consensus shutting down.
I20260619 07:01:21.085860 28130 raft_consensus.cc:2258] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:01:21.086225 28129 tablet_replica.cc:330] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: stopping tablet replica
I20260619 07:01:21.086580 28129 raft_consensus.cc:2229] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:01:21.086937 28129 raft_consensus.cc:2258] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:01:21.087603 28130 ts_tablet_manager.cc:1905] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:01:21.088582 28129 ts_tablet_manager.cc:1905] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 07:01:21.090958 27625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:38839 (error 108)
I20260619 07:01:21.091221 28129 ts_tablet_manager.cc:1918] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:01:21.091256 28130 ts_tablet_manager.cc:1918] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:01:21.091300 28129 log.cc:1196] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/02b25e789f5145ce993abd2df8954286
I20260619 07:01:21.091331 28130 log.cc:1196] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/02b25e789f5145ce993abd2df8954286
I20260619 07:01:21.091639 28129 ts_tablet_manager.cc:1939] T 02b25e789f5145ce993abd2df8954286 P 36f2522d45ce41a9b29b5af9a802d0f9: Deleting consensus metadata
I20260619 07:01:21.091646 28130 ts_tablet_manager.cc:1939] T 02b25e789f5145ce993abd2df8954286 P b4a78f36fcfb4086a6aa5c0dd0a21279: Deleting consensus metadata
I20260619 07:01:21.092368 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 27830
I20260619 07:01:21.096501 27625 catalog_manager.cc:4841] TS 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955): tablet 02b25e789f5145ce993abd2df8954286 (table test-table [id=fb9155c09af746869cc2b0cf47e4cbaa]) successfully deleted
I20260619 07:01:21.096735 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:21.096355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:38839 (local address 127.0.0.1:55298)
0619 07:01:21.096523 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:21.096630 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:38839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 07:01:21.096923 27625 catalog_manager.cc:4584] TS 72187e72c72a4c68897d702fb2e8c54b (127.11.106.65:38839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02b25e789f5145ce993abd2df8954286: Network error: Client connection negotiation failed: client connection to 127.11.106.65:38839: connect: Connection refused (error 111)
I20260619 07:01:21.101154 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 27968
I20260619 07:01:21.102408 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:21.102118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:41001 (local address 127.0.0.1:53580)
0619 07:01:21.102247 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:21.102312 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.66:41001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 07:01:21.107566 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 27605
I20260619 07:01:41.114893 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38655
--webserver_interface=127.11.106.126
--webserver_port=41631
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:38655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:01:41.224368 28132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:41.224704 28132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:41.224768 28132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:41.228770 28132 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 07:01:41.228826 28132 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 07:01:41.228842 28132 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 07:01:41.228855 28132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:01:41.228873 28132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:41.228938 28132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:41.228966 28132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:01:41.229007 28132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:01:41.234364 28132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:38655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=41631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:41.235468 28132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:41.236637 28132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:41.245082 28138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:41.245100 28137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:41.245100 28140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:41.245158 28132 server_base.cc:1034] running on GCE node
I20260619 07:01:41.245908 28132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:41.246968 28132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:41.248138 28132 hybrid_clock.cc:648] HybridClock initialized: now 1781852501248125 us; error 26 us; skew 500 ppm
Jun 19 07:01:41 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:41.250885 28132 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:01:41.251729 28132 webserver.cc:460] Webserver started at http://127.11.106.126:41631/ using document root <none> and password file <none>
I20260619 07:01:41.252267 28132 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:41.252326 28132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:41.256131 28132 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:01:41.258247 28147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:41.259469 28132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:01:41.259594 28132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "93c478b22a0342a28a7152e2206fabc0"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "24eaee16d069da5a8985551b97854d74"
server_key_iv: "cbb7b53b3defcffef1aaa201a0eec871"
server_key_version: "encryptionkey@0"
I20260619 07:01:41.260130 28132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:41.299778 28132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:41.302969 28132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:41.303139 28132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:41.310091 28132 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:38655
I20260619 07:01:41.310367 28199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:38655 every 8 connection(s)
I20260619 07:01:41.311255 28132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:01:41.311652 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 28132
I20260619 07:01:41.312081 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:38839
--local_ip_for_outbound_sockets=127.11.106.65
--tserver_master_addrs=127.11.106.126:38655
--webserver_port=39103
--webserver_interface=127.11.106.65
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:01:41.315794 28200 sys_catalog.cc:263] Verifying existing consensus state
I20260619 07:01:41.317855 28200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Bootstrap starting.
I20260619 07:01:41.335594 28200 log.cc:824] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Log is configured to *not* fsync() on all Append() calls
I20260619 07:01:41.347072 28200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 07:01:41.347522 28200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Bootstrap complete.
I20260619 07:01:41.351568 28200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:01:41.351977 28200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c478b22a0342a28a7152e2206fabc0, State: Initialized, Role: FOLLOWER
I20260619 07:01:41.352530 28200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:01:41.352733 28200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:01:41.352804 28200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:01:41.352898 28200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 2 FOLLOWER]: Advancing to term 3
I20260619 07:01:41.354774 28200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:01:41.355105 28200 leader_election.cc:302] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93c478b22a0342a28a7152e2206fabc0; no voters:
I20260619 07:01:41.355454 28200 leader_election.cc:288] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 07:01:41.355536 28204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 07:01:41.355784 28204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [term 3 LEADER]: Becoming Leader. State: Replica: 93c478b22a0342a28a7152e2206fabc0, State: Running, Role: LEADER
I20260619 07:01:41.356230 28204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } }
I20260619 07:01:41.356773 28200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:01:41.357273 28206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93c478b22a0342a28a7152e2206fabc0. Latest consensus state: current_term: 3 leader_uuid: "93c478b22a0342a28a7152e2206fabc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } } }
I20260619 07:01:41.357388 28206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: This master's current role is: LEADER
I20260619 07:01:41.357669 28205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "93c478b22a0342a28a7152e2206fabc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c478b22a0342a28a7152e2206fabc0" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38655 } } }
I20260619 07:01:41.357788 28205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0 [sys.catalog]: This master's current role is: LEADER
I20260619 07:01:41.357954 28209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:01:41.363070 28209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:01:41.364030 28209 catalog_manager.cc:1259] Loaded cluster ID: bdbef68fcc5845d08e755308da02fd41
I20260619 07:01:41.364090 28209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:01:41.366680 28209 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:01:41.367432 28209 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Loaded TSK: 0
I20260619 07:01:41.367515 28209 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93c478b22a0342a28a7152e2206fabc0: Loaded TSK: 1
F20260619 07:01:41.368280 28209 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f954e8e09ad google::LogMessage::Fail() at ??:0
@ 0x7f954e8e388a google::LogMessage::SendToLog() at ??:0
@ 0x7f954e8e04de google::LogMessage::Flush() at ??:0
@ 0x7f954e8e1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f95515a418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f955159cf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f95592457bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f955921f99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9559220178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f9559261df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f955928082b std::function<>::operator()() at ??:0
@ 0x7f955921fb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9559220f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f955921d579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9559261640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9550b323d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9550b32c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9550b343f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9550b23512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f95521676db start_thread at ??:0
@ 0x7f954d3de61f clone at ??:0
W20260619 07:01:41.464303 28202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:41.464707 28202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:41.464790 28202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:41.471130 28202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:41.471204 28202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:41.471325 28202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:01:41.479982 28202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:38839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=39103
--tserver_master_addrs=127.11.106.126:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:41.481407 28202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:41.482846 28202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:41.494328 28247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:41.494323 28248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:41.494390 28250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:41.494912 28202 server_base.cc:1034] running on GCE node
I20260619 07:01:41.495296 28202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:41.495942 28202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:41.497157 28202 hybrid_clock.cc:648] HybridClock initialized: now 1781852501497125 us; error 49 us; skew 500 ppm
Jun 19 07:01:41 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:41.500151 28202 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:01:41.501058 28202 webserver.cc:460] Webserver started at http://127.11.106.65:39103/ using document root <none> and password file <none>
I20260619 07:01:41.501652 28202 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:41.501745 28202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:41.505726 28202 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:01:41.508056 28257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:41.509677 28202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 07:01:41.509829 28202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "72187e72c72a4c68897d702fb2e8c54b"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "b6d13ce2de93b182ed3152a05db721bf"
server_key_iv: "c5f5fca4e50432546a877b943c219373"
server_key_version: "encryptionkey@0"
I20260619 07:01:41.510421 28202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:41.524509 28202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:41.527920 28202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:41.528129 28202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:41.529001 28202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:41.530117 28264 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 07:01:41.535792 28202 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 07:01:41.535885 28202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 07:01:41.535967 28202 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 07:01:41.537899 28202 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 07:01:41.537962 28202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 07:01:41.538298 28264 tablet_bootstrap.cc:492] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Bootstrap starting.
I20260619 07:01:41.548645 28202 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:38839
I20260619 07:01:41.548808 28371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:38839 every 8 connection(s)
I20260619 07:01:41.549713 28202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:01:41.559365 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 28202
I20260619 07:01:41.559818 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:41001
--local_ip_for_outbound_sockets=127.11.106.66
--tserver_master_addrs=127.11.106.126:38655
--webserver_port=43571
--webserver_interface=127.11.106.66
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:01:41.560470 28264 log.cc:824] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Log is configured to *not* fsync() on all Append() calls
I20260619 07:01:41.567462 28264 tablet_bootstrap.cc:492] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 07:01:41.567840 28264 tablet_bootstrap.cc:492] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Bootstrap complete.
I20260619 07:01:41.568989 28264 ts_tablet_manager.cc:1397] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20260619 07:01:41.571298 28264 raft_consensus.cc:348] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:41.571715 28264 raft_consensus.cc:729] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72187e72c72a4c68897d702fb2e8c54b, State: Initialized, Role: FOLLOWER
I20260619 07:01:41.572211 28264 consensus_queue.cc:260] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:41.573169 28264 ts_tablet_manager.cc:1428] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260619 07:01:41.634624 28374 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:01:41.551686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:45359)
0619 07:01:41.552177 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:41.552230 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:41.553062 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:41.633652 (+ 80590us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":143,"threads_started":1}
W20260619 07:01:41.635291 28372 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:38655 (0 consecutive failures): Network error: Failed to ping master at 127.11.106.126:38655: Client connection negotiation failed: client connection to 127.11.106.126:38655: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 07:01:41.636039 28374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:41.635763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:60983)
0619 07:01:41.635902 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:41.635976 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 07:01:41.636806 28374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:41.636589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:58789)
0619 07:01:41.636701 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:41.636749 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 07:01:41.637017 28372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 07:01:41.685607 28376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:41.685853 28376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:41.685894 28376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:41.689739 28376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:41.689787 28376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:41.689841 28376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:01:41.694952 28376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:41001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=43571
--tserver_master_addrs=127.11.106.126:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:41.695884 28376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:41.696940 28376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:41.706426 28382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:41.706427 28383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:41.706578 28376 server_base.cc:1034] running on GCE node
W20260619 07:01:41.706794 28385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:41.707177 28376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:41.707855 28376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:41.709048 28376 hybrid_clock.cc:648] HybridClock initialized: now 1781852501709029 us; error 30 us; skew 500 ppm
Jun 19 07:01:41 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:41.711802 28376 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:01:41.712759 28376 webserver.cc:460] Webserver started at http://127.11.106.66:43571/ using document root <none> and password file <none>
I20260619 07:01:41.713387 28376 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:41.713487 28376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:41.717484 28376 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:01:41.719497 28392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:41.720834 28376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:01:41.721021 28376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "87b0b40567eb76d085a220d6b485ce83"
server_key_iv: "ee74407e502732390ba32d708dd2e90e"
server_key_version: "encryptionkey@0"
I20260619 07:01:41.721578 28376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:41.745893 28376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:41.748993 28376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:41.749207 28376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:41.749854 28376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:41.750723 28376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:01:41.750795 28376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:41.750867 28376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:01:41.750911 28376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:41.760166 28376 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:41001
I20260619 07:01:41.760190 28505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:41001 every 8 connection(s)
I20260619 07:01:41.761257 28376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:01:41.765041 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:41.763409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:47011)
0619 07:01:41.764104 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:41.764214 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":123,"threads_started":1}
W20260619 07:01:41.765479 28506 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:38655 (0 consecutive failures): Network error: Failed to ping master at 127.11.106.126:38655: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
I20260619 07:01:41.766095 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:41.765857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:44105)
0619 07:01:41.765973 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:41.766025 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 07:01:41.766659 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:41.766450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:49307)
0619 07:01:41.766565 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:41.766606 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 07:01:41.766839 28506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 07:01:41.767347 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 28376
I20260619 07:01:41.767781 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:43955
--local_ip_for_outbound_sockets=127.11.106.67
--tserver_master_addrs=127.11.106.126:38655
--webserver_port=42227
--webserver_interface=127.11.106.67
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:01:41.876379 28509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:41.876713 28509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:41.876760 28509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:41.880645 28509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:41.880692 28509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:41.880748 28509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:01:41.885686 28509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:43955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=42227
--tserver_master_addrs=127.11.106.126:38655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:41.886622 28509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:41.887589 28509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:41.897281 28515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:41.897303 28514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:41.897348 28517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:41.897457 28509 server_base.cc:1034] running on GCE node
I20260619 07:01:41.897938 28509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:41.898587 28509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:41.899746 28509 hybrid_clock.cc:648] HybridClock initialized: now 1781852501899713 us; error 51 us; skew 500 ppm
Jun 19 07:01:41 dist-test-slave-10d0 krb5kdc[27057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:41.902415 28509 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:01:41.903290 28509 webserver.cc:460] Webserver started at http://127.11.106.67:42227/ using document root <none> and password file <none>
I20260619 07:01:41.903843 28509 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:41.903903 28509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:41.907652 28509 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:01:41.909713 28524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:41.910934 28509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:01:41.911068 28509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "36f2522d45ce41a9b29b5af9a802d0f9"
format_stamp: "Formatted at 2026-06-19 07:00:59 on dist-test-slave-10d0"
server_key: "7c5136418f4027a9e20750e8b8e407bc"
server_key_iv: "e3aaad4d06cae2dcffed9437566ad048"
server_key_version: "encryptionkey@0"
I20260619 07:01:41.911615 28509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:41.927589 28509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:41.930694 28509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:41.930910 28509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:41.931524 28509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:41.932370 28509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:01:41.932441 28509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:41.932512 28509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:01:41.932556 28509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:41.941977 28509 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:43955
I20260619 07:01:41.942010 28637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:43955 every 8 connection(s)
I20260619 07:01:41.942981 28509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:01:41.944388 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 28509
I20260619 07:01:41.946868 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:41.945360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:49173)
0619 07:01:41.946050 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:41.946150 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":137,"threads_started":1}
W20260619 07:01:41.947378 28638 heartbeater.cc:644] Failed to heartbeat to 127.11.106.126:38655 (0 consecutive failures): Network error: Failed to ping master at 127.11.106.126:38655: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
I20260619 07:01:41.947971 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:41.947767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:53433)
0619 07:01:41.947877 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:41.947919 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 07:01:41.948585 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:41.948376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:37229)
0619 07:01:41.948464 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:41.948501 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 07:01:41.948762 28638 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 07:01:42.638213 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:42.637682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:51117)
0619 07:01:42.638036 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:42.638097 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":98,"threads_started":1}
I20260619 07:01:42.767818 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:42.767438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:50303)
0619 07:01:42.767672 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:42.767722 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260619 07:01:42.949874 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:42.949419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:47483)
0619 07:01:42.949681 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:42.949744 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260619 07:01:42.959028 28645 raft_consensus.cc:482] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:01:42.959223 28645 raft_consensus.cc:504] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:42.960263 28645 leader_election.cc:288] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955), b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001)
Jun 19 07:01:42 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.67@KRBTEST.COM
Jun 19 07:01:42 dist-test-slave-10d0 krb5kdc[27057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852501, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.66@KRBTEST.COM
I20260619 07:01:42.974738 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:42.960659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:40089 (local address 127.11.106.67:43955)
0619 07:01:42.961058 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:42.961088 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:42.961140 (+ 52us) server_negotiation.cc:407] Connection header received
0619 07:01:42.961863 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:42.961869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:42.962221 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:42.962470 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:42.963852 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:42.964882 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:42.966032 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:42.966502 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:42.966696 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:42.969492 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:42.969496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:42.969508 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:42.969535 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:42.971422 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:42.972174 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:42.972180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:42.972186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:42.972345 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:42.972819 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:42.972822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:42.972824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:42.973327 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:42.973464 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:42.974223 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 07:01:42.974532 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
I20260619 07:01:42.974738 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:42.961227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:42755 (local address 127.11.106.66:41001)
0619 07:01:42.961541 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:42.961569 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:42.961598 (+ 29us) server_negotiation.cc:407] Connection header received
0619 07:01:42.962249 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:42.962255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:42.962643 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:42.962939 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:42.963852 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:42.964871 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:42.966070 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:42.966499 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:42.966696 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:42.969807 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:42.969811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:42.969823 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:42.969850 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:42.971433 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:42.972174 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:42.972180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:42.972186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:42.972345 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:42.972819 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:42.972822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:42.972824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:42.973327 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:42.973463 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:42.974223 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 07:01:42.974532 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":135,"threads_started":1}
I20260619 07:01:42.974766 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:42.960784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:41001 (local address 127.11.106.65:42755)
0619 07:01:42.961055 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:42.961094 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:42.961188 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:42.962894 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:42.962897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:42.963176 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:42.963676 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:42.963691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:42.965029 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:42.965043 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:42.965912 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:42.965922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:42.966628 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:42.966631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:42.966885 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:42.967557 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:01:42.967573 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:42.969612 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:42.971608 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:42.971612 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:42.971616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:42.972050 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:42.972452 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:42.972455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:42.972457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:42.972710 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:42.973450 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:42.973457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:42.973844 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 07:01:42.974065 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 07:01:42.974512 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":270,"spinlock_wait_cycles":27648,"thread_start_us":71,"threads_started":1}
I20260619 07:01:42.974768 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:42.960489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:43955 (local address 127.11.106.65:40089)
0619 07:01:42.960661 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:42.960676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:42.960795 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:42.962584 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:42.962591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:42.963196 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:42.963677 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:42.963691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:42.965037 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:42.965040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:42.965909 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:42.965918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:42.966666 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:42.966670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:42.966889 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:42.967614 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:01:42.967626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:42.969251 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:42.971551 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:42.971560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:42.971575 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:42.972050 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:42.972452 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:42.972455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:42.972457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:42.972709 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:42.973450 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:42.973457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:42.973844 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 07:01:42.974065 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 07:01:42.974563 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":327,"spinlock_wait_cycles":14848}
W20260619 07:01:42.977327 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
W20260619 07:01:42.977555 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
I20260619 07:01:42.977617 28261 leader_election.cc:302] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72187e72c72a4c68897d702fb2e8c54b; no voters: 36f2522d45ce41a9b29b5af9a802d0f9, b4a78f36fcfb4086a6aa5c0dd0a21279
I20260619 07:01:42.977911 28645 raft_consensus.cc:2726] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:01:43.639324 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:43.638940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:37201)
0619 07:01:43.639177 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:43.639230 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260619 07:01:43.769102 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:43.768520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:48565)
0619 07:01:43.768954 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:43.769012 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":77,"threads_started":1}
I20260619 07:01:43.951121 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:43.950705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:36483)
0619 07:01:43.950964 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:43.951020 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":82,"threads_started":1}
I20260619 07:01:44.640566 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:44.640133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:38035)
0619 07:01:44.640399 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:44.640452 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260619 07:01:44.770344 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:44.769831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:51539)
0619 07:01:44.770178 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:44.770233 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":88,"threads_started":1}
I20260619 07:01:44.952447 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:44.951882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:45691)
0619 07:01:44.952230 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:44.952310 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":111,"threads_started":1}
I20260619 07:01:45.002626 28655 raft_consensus.cc:482] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:01:45.002759 28655 raft_consensus.cc:504] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:45.003235 28655 leader_election.cc:288] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955), b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001)
W20260619 07:01:45.004161 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
W20260619 07:01:45.004289 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
I20260619 07:01:45.004328 28261 leader_election.cc:302] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72187e72c72a4c68897d702fb2e8c54b; no voters: 36f2522d45ce41a9b29b5af9a802d0f9, b4a78f36fcfb4086a6aa5c0dd0a21279
I20260619 07:01:45.004470 28655 raft_consensus.cc:2726] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:01:45.641784 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:45.641384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:44489)
0619 07:01:45.641640 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:45.641692 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260619 07:01:45.771459 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:45.771097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:51021)
0619 07:01:45.771321 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:45.771371 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260619 07:01:45.953912 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:45.953296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:35693)
0619 07:01:45.953766 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:45.953823 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":87,"threads_started":1}
I20260619 07:01:46.643110 28659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:46.642638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:48843)
0619 07:01:46.642939 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:46.643013 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":69,"threads_started":1}
I20260619 07:01:46.672250 28660 raft_consensus.cc:482] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:01:46.672382 28660 raft_consensus.cc:504] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:46.672900 28660 leader_election.cc:288] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955), b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001)
W20260619 07:01:46.673725 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
W20260619 07:01:46.673890 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
I20260619 07:01:46.673944 28261 leader_election.cc:302] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72187e72c72a4c68897d702fb2e8c54b; no voters: 36f2522d45ce41a9b29b5af9a802d0f9, b4a78f36fcfb4086a6aa5c0dd0a21279
I20260619 07:01:46.674140 28660 raft_consensus.cc:2726] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:01:46.772716 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:46.772320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:36405)
0619 07:01:46.772571 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:46.772625 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260619 07:01:46.955120 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:46.954656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:57665)
0619 07:01:46.954977 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:46.955033 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":132,"threads_started":1}
I20260619 07:01:47.644249 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:47.643865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:42783)
0619 07:01:47.644105 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:47.644157 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260619 07:01:47.773923 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:47.773510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:52683)
0619 07:01:47.773776 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:47.773833 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260619 07:01:47.956264 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:47.955868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:40247)
0619 07:01:47.956118 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:47.956173 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260619 07:01:48.645450 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:48.644955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:51657)
0619 07:01:48.645297 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:48.645354 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":78,"threads_started":1}
I20260619 07:01:48.775101 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:48.774661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:35937)
0619 07:01:48.774957 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:48.775010 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
I20260619 07:01:48.957556 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:48.957098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:40433)
0619 07:01:48.957404 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:48.957461 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":88,"threads_started":1}
I20260619 07:01:49.646837 28669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:49.646336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:57295)
0619 07:01:49.646639 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:49.646698 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260619 07:01:49.776311 28670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:49.775863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:33165)
0619 07:01:49.776124 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:49.776182 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260619 07:01:49.958781 28671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:49.958355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:38715)
0619 07:01:49.958627 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:49.958689 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260619 07:01:50.648067 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:50.647602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:41005)
0619 07:01:50.647924 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:50.647973 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":87,"threads_started":1}
I20260619 07:01:50.777921 28673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:50.777406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:58647)
0619 07:01:50.777721 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:50.777784 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
I20260619 07:01:50.919529 28674 raft_consensus.cc:482] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:01:50.919679 28674 raft_consensus.cc:504] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:50.920109 28674 leader_election.cc:288] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955), b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001)
W20260619 07:01:50.920893 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
W20260619 07:01:50.921025 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
I20260619 07:01:50.921069 28261 leader_election.cc:302] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72187e72c72a4c68897d702fb2e8c54b; no voters: 36f2522d45ce41a9b29b5af9a802d0f9, b4a78f36fcfb4086a6aa5c0dd0a21279
I20260619 07:01:50.921202 28674 raft_consensus.cc:2726] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:01:50.959915 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:50.959525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:40137)
0619 07:01:50.959769 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:50.959825 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260619 07:01:51.649528 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:51.649106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:40953)
0619 07:01:51.649376 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:51.649432 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
I20260619 07:01:51.779134 28677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:51.778736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:50813)
0619 07:01:51.778979 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:51.779035 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260619 07:01:51.961347 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:51.960919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:39471)
0619 07:01:51.961189 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:51.961251 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260619 07:01:52.650743 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:52.650343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:46817)
0619 07:01:52.650591 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:52.650644 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260619 07:01:52.780309 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:52.779894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:40793)
0619 07:01:52.780161 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:52.780216 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260619 07:01:52.962486 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:52.962092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:50545)
0619 07:01:52.962342 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:52.962395 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260619 07:01:53.651978 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:53.651494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:42589)
0619 07:01:53.651818 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:53.651874 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":126,"threads_started":1}
I20260619 07:01:53.781528 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:53.781128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:58485)
0619 07:01:53.781377 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:53.781433 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260619 07:01:53.963708 28684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:53.963316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:52771)
0619 07:01:53.963543 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:53.963614 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260619 07:01:54.653215 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:54.652789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:57475)
0619 07:01:54.653044 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:54.653117 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
I20260619 07:01:54.782774 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:54.782372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:57293)
0619 07:01:54.782624 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:54.782680 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260619 07:01:54.964910 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:54.964468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:51841)
0619 07:01:54.964769 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:54.964819 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":140,"threads_started":1}
I20260619 07:01:54.986373 28688 raft_consensus.cc:482] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:01:54.986500 28688 raft_consensus.cc:504] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f2522d45ce41a9b29b5af9a802d0f9" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 43955 } } peers { permanent_uuid: "b4a78f36fcfb4086a6aa5c0dd0a21279" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 41001 } } peers { permanent_uuid: "72187e72c72a4c68897d702fb2e8c54b" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 38839 } }
I20260619 07:01:54.986979 28688 leader_election.cc:288] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955), b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001)
W20260619 07:01:54.987802 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4a78f36fcfb4086a6aa5c0dd0a21279 (127.11.106.66:41001): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
W20260619 07:01:54.987934 28261 leader_election.cc:341] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 36f2522d45ce41a9b29b5af9a802d0f9 (127.11.106.67:43955): Not found: Tablet not found: 02b25e789f5145ce993abd2df8954286
I20260619 07:01:54.987993 28261 leader_election.cc:302] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72187e72c72a4c68897d702fb2e8c54b; no voters: 36f2522d45ce41a9b29b5af9a802d0f9, b4a78f36fcfb4086a6aa5c0dd0a21279
I20260619 07:01:54.988158 28688 raft_consensus.cc:2726] T 02b25e789f5145ce993abd2df8954286 P 72187e72c72a4c68897d702fb2e8c54b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 07:01:55.654408 28689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:55.653979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:35773)
0619 07:01:55.654256 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:55.654312 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
I20260619 07:01:55.783931 28690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:55.783508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:48011)
0619 07:01:55.783781 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:55.783836 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":105,"threads_started":1}
I20260619 07:01:55.966233 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:55.965739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.67:45929)
0619 07:01:55.966077 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:55.966137 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":76,"threads_started":1}
I20260619 07:01:56.655800 28692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:56.655396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.65:52503)
0619 07:01:56.655651 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:56.655705 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260619 07:01:56.785269 28693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:56.784877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:38655 (local address 127.11.106.66:46573)
0619 07:01:56.785116 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:56.785173 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.126:38655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260619 07:01:56.951355 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 28202
I20260619 07:01:56.957590 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 28376
I20260619 07:01:56.963020 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 28509
2026-06-19T07:01:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59230 ms)
[----------] 30 tests from SecurityITest (243468 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:01:57 dist-test-slave-10d0 krb5kdc[28696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:01:57 dist-test-slave-10d0 krb5kdc[28696](info): set up 2 sockets
Jun 19 07:01:57 dist-test-slave-10d0 krb5kdc[28696](info): commencing operation
krb5kdc: starting...
W20260619 07:01:58.053015 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:01:58 dist-test-slave-10d0 krb5kdc[28696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:01:58.175976 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:39389
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:39389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:01:58.292760 28709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:58.293093 28709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:58.293180 28709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:58.297349 28709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:01:58.297437 28709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:58.297477 28709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:58.297534 28709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:01:58.297572 28709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:01:58.302937 28709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:39389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:39389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:58.304067 28709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:58.305227 28709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:58.314253 28717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:58.314288 28715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:58.314330 28709 server_base.cc:1034] running on GCE node
W20260619 07:01:58.314571 28714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:58.315100 28709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:58.316210 28709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:58.317368 28709 hybrid_clock.cc:648] HybridClock initialized: now 1781852518317353 us; error 26 us; skew 500 ppm
Jun 19 07:01:58 dist-test-slave-10d0 krb5kdc[28696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852518, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:58.320214 28709 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:01:58.321241 28709 webserver.cc:460] Webserver started at http://127.11.106.126:34635/ using document root <none> and password file <none>
I20260619 07:01:58.321844 28709 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:58.321941 28709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:58.322177 28709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:01:58.324229 28709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "0e0aba092f5948619e5b615db21e0997"
format_stamp: "Formatted at 2026-06-19 07:01:58 on dist-test-slave-10d0"
server_key: "608eb4e07fc21db10396393b5463eacb"
server_key_iv: "6a4a4142ccc933a7aa311b40fba47af4"
server_key_version: "encryptionkey@0"
I20260619 07:01:58.324819 28709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "0e0aba092f5948619e5b615db21e0997"
format_stamp: "Formatted at 2026-06-19 07:01:58 on dist-test-slave-10d0"
server_key: "608eb4e07fc21db10396393b5463eacb"
server_key_iv: "6a4a4142ccc933a7aa311b40fba47af4"
server_key_version: "encryptionkey@0"
I20260619 07:01:58.328471 28709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 07:01:58.330952 28724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:58.332170 28709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 07:01:58.332320 28709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "0e0aba092f5948619e5b615db21e0997"
format_stamp: "Formatted at 2026-06-19 07:01:58 on dist-test-slave-10d0"
server_key: "608eb4e07fc21db10396393b5463eacb"
server_key_iv: "6a4a4142ccc933a7aa311b40fba47af4"
server_key_version: "encryptionkey@0"
I20260619 07:01:58.332434 28709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:58.346051 28709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:58.348917 28709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:58.349115 28709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:58.356099 28709 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:39389
I20260619 07:01:58.356127 28776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:39389 every 8 connection(s)
I20260619 07:01:58.357088 28709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:01:58.359882 28777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:01:58.362344 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 28709
I20260619 07:01:58.362483 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:01:58.362797 11689 external_mini_cluster.cc:1428] Setting key 4aa49eca55e8379b29bc13117e49c0e1
I20260619 07:01:58.366091 28777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997: Bootstrap starting.
I20260619 07:01:58.368710 28777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997: Neither blocks nor log segments found. Creating new log.
I20260619 07:01:58.369554 28777 log.cc:824] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997: Log is configured to *not* fsync() on all Append() calls
I20260619 07:01:58.371636 28777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997: No bootstrap required, opened a new log
Jun 19 07:01:58 dist-test-slave-10d0 krb5kdc[28696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:01:58.375097 28777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0aba092f5948619e5b615db21e0997" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39389 } }
I20260619 07:01:58.375375 28777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:01:58.375465 28777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0aba092f5948619e5b615db21e0997, State: Initialized, Role: FOLLOWER
I20260619 07:01:58.376035 28777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0aba092f5948619e5b615db21e0997" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39389 } }
I20260619 07:01:58.376215 28777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:01:58.376287 28777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:01:58.376394 28777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:01:58.377527 28777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0aba092f5948619e5b615db21e0997" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39389 } }
I20260619 07:01:58.377892 28777 leader_election.cc:302] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e0aba092f5948619e5b615db21e0997; no voters:
I20260619 07:01:58.378065 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:58.364277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.11.106.126:39389)
0619 07:01:58.364747 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:58.364762 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:58.364799 (+ 37us) server_negotiation.cc:407] Connection header received
0619 07:01:58.365444 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:58.365462 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:58.365797 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:58.366084 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:58.366999 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.367990 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.369244 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.369867 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.370124 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:58.372640 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:58.372646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:58.372659 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:58.372686 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:58.374928 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:58.375390 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:58.375398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:58.375409 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:58.375628 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:58.375934 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:58.375938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:58.375940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:58.376468 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:58.376709 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:58.376952 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 07:01:58.377246 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":134,"threads_started":1}
I20260619 07:01:58.378262 28777 leader_election.cc:288] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:01:58.378351 28782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:01:58.378579 28782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [term 1 LEADER]: Becoming Leader. State: Replica: 0e0aba092f5948619e5b615db21e0997, State: Running, Role: LEADER
I20260619 07:01:58.378931 28782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0aba092f5948619e5b615db21e0997" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39389 } }
I20260619 07:01:58.379686 28777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:01:58.381103 28783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e0aba092f5948619e5b615db21e0997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0aba092f5948619e5b615db21e0997" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39389 } } }
I20260619 07:01:58.381449 28783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [sys.catalog]: This master's current role is: LEADER
I20260619 07:01:58.381731 28784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e0aba092f5948619e5b615db21e0997. Latest consensus state: current_term: 1 leader_uuid: "0e0aba092f5948619e5b615db21e0997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e0aba092f5948619e5b615db21e0997" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 39389 } } }
I20260619 07:01:58.381892 28784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997 [sys.catalog]: This master's current role is: LEADER
I20260619 07:01:58.381963 28795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:01:58.385308 28795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:01:58.391207 28795 catalog_manager.cc:1347] Generated new cluster ID: 090fbecaa870485a859075436f20e6d2
I20260619 07:01:58.391284 28795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:01:58.408242 28795 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:01:58.409581 28795 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:01:58.419986 28795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997: Generated new TSK 0
I20260619 07:01:58.420814 28795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 07:01:58.448491 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39389
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:01:58.562327 28803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:58.562582 28803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:58.562626 28803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:58.566632 28803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:58.566692 28803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:58.566751 28803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:01:58.571808 28803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:58.572844 28803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:58.573873 28803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:58.583636 28808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:58.583781 28809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:58.583781 28811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:58.583912 28803 server_base.cc:1034] running on GCE node
I20260619 07:01:58.584362 28803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:58.585047 28803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:58.586208 28803 hybrid_clock.cc:648] HybridClock initialized: now 1781852518586177 us; error 45 us; skew 500 ppm
Jun 19 07:01:58 dist-test-slave-10d0 krb5kdc[28696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852518, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:58.588832 28803 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:01:58.589702 28803 webserver.cc:460] Webserver started at http://127.11.106.65:41883/ using document root <none> and password file <none>
I20260619 07:01:58.590267 28803 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:58.590358 28803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:58.590586 28803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:01:58.592605 28803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "fb15badc63124b8aa5c558d5fb08f318"
format_stamp: "Formatted at 2026-06-19 07:01:58 on dist-test-slave-10d0"
server_key: "ad432804481c8bca13a94325e3077132"
server_key_iv: "5be050745e7169cafc8cace8c0925c5e"
server_key_version: "encryptionkey@0"
I20260619 07:01:58.593159 28803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "fb15badc63124b8aa5c558d5fb08f318"
format_stamp: "Formatted at 2026-06-19 07:01:58 on dist-test-slave-10d0"
server_key: "ad432804481c8bca13a94325e3077132"
server_key_iv: "5be050745e7169cafc8cace8c0925c5e"
server_key_version: "encryptionkey@0"
I20260619 07:01:58.596863 28803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:01:58.599206 28818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:58.600379 28803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:01:58.600518 28803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "fb15badc63124b8aa5c558d5fb08f318"
format_stamp: "Formatted at 2026-06-19 07:01:58 on dist-test-slave-10d0"
server_key: "ad432804481c8bca13a94325e3077132"
server_key_iv: "5be050745e7169cafc8cace8c0925c5e"
server_key_version: "encryptionkey@0"
I20260619 07:01:58.600679 28803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:58.616225 28803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:58.619556 28803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:58.619758 28803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:58.620357 28803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:58.621191 28803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:01:58.621263 28803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:58.621333 28803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:01:58.621384 28803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:58.630621 28803 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:46015
I20260619 07:01:58.630661 28931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:46015 every 8 connection(s)
I20260619 07:01:58.631553 28803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:01:58.634822 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 28803
I20260619 07:01:58.634924 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 07:01:58.635214 11689 external_mini_cluster.cc:1428] Setting key 8769022e6236a1e03983690fc92d5b18
Jun 19 07:01:58 dist-test-slave-10d0 krb5kdc[28696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852518, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 07:01:58.647092 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:58.633310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:36233 (local address 127.11.106.126:39389)
0619 07:01:58.633478 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:58.633483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:58.634354 (+ 871us) server_negotiation.cc:407] Connection header received
0619 07:01:58.636107 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:58.636110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:58.636152 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:58.636225 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:58.637892 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.638484 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.639772 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.640202 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.640322 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:58.643234 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:58.643238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:58.643242 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:58.643274 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:58.645021 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:58.645510 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:58.645516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:58.645518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:58.645630 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:58.645981 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:58.645985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:58.645986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:58.646168 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:58.646249 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:58.646773 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 07:01:58.646909 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 07:01:58.648010 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:58.633719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39389 (local address 127.11.106.65:36233)
0619 07:01:58.634187 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:58.634232 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:58.635858 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:58.636410 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:58.636430 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:58.636959 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:58.637713 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:58.637738 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.638660 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.638665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:58.639643 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:58.639652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.640352 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.640354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:58.640614 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:58.641302 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:01:58.641320 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:58.643048 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:58.645140 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:58.645145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:58.645158 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:58.645409 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:58.645746 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:58.645748 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:58.645749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:58.645883 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:58.646261 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:58.646266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:58.646493 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 07:01:58.646722 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 07:01:58.647047 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 07:01:58.649506 28932 heartbeater.cc:347] Connected to a master server at 127.11.106.126:39389
I20260619 07:01:58.649780 28932 heartbeater.cc:464] Registering TS with master...
I20260619 07:01:58.650345 28932 heartbeater.cc:511] Master 127.11.106.126:39389 requested a full tablet report, sending...
I20260619 07:01:58.651999 28741 ts_manager.cc:194] Registered new tserver with Master: fb15badc63124b8aa5c558d5fb08f318 (127.11.106.65:46015)
I20260619 07:01:58.653213 28741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:36233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 07:01:58.666023 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39389
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:01:58.778633 28937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:58.778908 28937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:58.778995 28937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:58.783115 28937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:58.783200 28937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:58.783289 28937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:01:58.788589 28937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:58.789590 28937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:58.790596 28937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:58.800060 28945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:58.800118 28942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:58.800154 28943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:58.800113 28937 server_base.cc:1034] running on GCE node
I20260619 07:01:58.800787 28937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:58.801467 28937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:58.802654 28937 hybrid_clock.cc:648] HybridClock initialized: now 1781852518802639 us; error 27 us; skew 500 ppm
Jun 19 07:01:58 dist-test-slave-10d0 krb5kdc[28696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852518, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:58.805487 28937 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:01:58.806404 28937 webserver.cc:460] Webserver started at http://127.11.106.66:43245/ using document root <none> and password file <none>
I20260619 07:01:58.806972 28937 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:58.807062 28937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:58.807294 28937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:01:58.809401 28937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "64a81db2dba0489e9b64f35e2a7bf6a2"
format_stamp: "Formatted at 2026-06-19 07:01:58 on dist-test-slave-10d0"
server_key: "b85c262968fa86dd9d4449ca88c87798"
server_key_iv: "1aab0e3517b8fa15fdf31a63035f100c"
server_key_version: "encryptionkey@0"
I20260619 07:01:58.809971 28937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "64a81db2dba0489e9b64f35e2a7bf6a2"
format_stamp: "Formatted at 2026-06-19 07:01:58 on dist-test-slave-10d0"
server_key: "b85c262968fa86dd9d4449ca88c87798"
server_key_iv: "1aab0e3517b8fa15fdf31a63035f100c"
server_key_version: "encryptionkey@0"
I20260619 07:01:58.813716 28937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 07:01:58.816042 28952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:58.817299 28937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 07:01:58.817438 28937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "64a81db2dba0489e9b64f35e2a7bf6a2"
format_stamp: "Formatted at 2026-06-19 07:01:58 on dist-test-slave-10d0"
server_key: "b85c262968fa86dd9d4449ca88c87798"
server_key_iv: "1aab0e3517b8fa15fdf31a63035f100c"
server_key_version: "encryptionkey@0"
I20260619 07:01:58.817561 28937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:58.830297 28937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:58.833638 28937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:58.833843 28937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:58.834442 28937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:58.835395 28937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:01:58.835464 28937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:58.835533 28937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:01:58.835592 28937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:58.844529 28937 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:45009
I20260619 07:01:58.844617 29065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:45009 every 8 connection(s)
I20260619 07:01:58.845484 28937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:01:58.852072 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 28937
I20260619 07:01:58.852224 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 07:01:58.852587 11689 external_mini_cluster.cc:1428] Setting key 92760c0342d0acf7b76e63e0a2e25db2
Jun 19 07:01:58 dist-test-slave-10d0 krb5kdc[28696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852518, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:01:58.860319 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:58.847279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:45859 (local address 127.11.106.126:39389)
0619 07:01:58.847420 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:58.847423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:58.848258 (+ 835us) server_negotiation.cc:407] Connection header received
0619 07:01:58.849654 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:58.849658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:58.849716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:58.849816 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:58.850984 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.851634 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.853297 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.853528 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.853649 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:58.856433 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:58.856440 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:58.856445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:58.856478 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:58.857971 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:58.858656 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:58.858659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:58.858660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:58.858734 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:58.859139 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:58.859141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:58.859142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:58.859312 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:58.859385 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:58.860098 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 07:01:58.860199 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 07:01:58.861603 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:58.847656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39389 (local address 127.11.106.66:45859)
0619 07:01:58.848106 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:58.848150 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:58.849430 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:58.849993 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:58.850000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:58.850350 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:58.850837 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:58.850851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.851773 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.851776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:58.853164 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:58.853174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:58.853687 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:58.853691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:58.853860 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:58.854461 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:01:58.854486 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:58.856258 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:58.858098 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:58.858107 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:58.858124 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:58.858532 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:58.858829 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:58.858833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:58.858838 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:58.859037 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:58.859420 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:58.859430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:58.859806 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 07:01:58.860054 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 07:01:58.860438 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 07:01:58.862794 29066 heartbeater.cc:347] Connected to a master server at 127.11.106.126:39389
I20260619 07:01:58.863054 29066 heartbeater.cc:464] Registering TS with master...
I20260619 07:01:58.863540 29066 heartbeater.cc:511] Master 127.11.106.126:39389 requested a full tablet report, sending...
I20260619 07:01:58.864666 28741 ts_manager.cc:194] Registered new tserver with Master: 64a81db2dba0489e9b64f35e2a7bf6a2 (127.11.106.66:45009)
I20260619 07:01:58.865258 28741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:45859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 07:01:58.879878 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39389
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:44897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:01:58.993057 29071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:58.993309 29071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:58.993352 29071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:58.997567 29071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:58.997627 29071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:58.997686 29071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:01:59.002893 29071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:44897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:39389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:59.003878 29071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:59.005143 29071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:59.014880 29079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:59.014891 29076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:59.014919 29077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:59.015133 29071 server_base.cc:1034] running on GCE node
I20260619 07:01:59.015635 29071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:59.016304 29071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:59.017488 29071 hybrid_clock.cc:648] HybridClock initialized: now 1781852519017450 us; error 48 us; skew 500 ppm
Jun 19 07:01:59 dist-test-slave-10d0 krb5kdc[28696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852519, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:01:59.020241 29071 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:01:59.021205 29071 webserver.cc:460] Webserver started at http://127.11.106.67:35583/ using document root <none> and password file <none>
I20260619 07:01:59.021816 29071 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:59.021905 29071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:59.022147 29071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:01:59.024183 29071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "318c85877a8a409cbed3fed81d2dbd59"
format_stamp: "Formatted at 2026-06-19 07:01:59 on dist-test-slave-10d0"
server_key: "69e2c922907071e0ab4d42b0f345ef06"
server_key_iv: "faf18c3b04ef4f5b7e5c47deda041683"
server_key_version: "encryptionkey@0"
I20260619 07:01:59.024776 29071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "318c85877a8a409cbed3fed81d2dbd59"
format_stamp: "Formatted at 2026-06-19 07:01:59 on dist-test-slave-10d0"
server_key: "69e2c922907071e0ab4d42b0f345ef06"
server_key_iv: "faf18c3b04ef4f5b7e5c47deda041683"
server_key_version: "encryptionkey@0"
I20260619 07:01:59.028476 29071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 07:01:59.030860 29086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:59.032163 29071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 07:01:59.032255 29071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "318c85877a8a409cbed3fed81d2dbd59"
format_stamp: "Formatted at 2026-06-19 07:01:59 on dist-test-slave-10d0"
server_key: "69e2c922907071e0ab4d42b0f345ef06"
server_key_iv: "faf18c3b04ef4f5b7e5c47deda041683"
server_key_version: "encryptionkey@0"
I20260619 07:01:59.032338 29071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:59.059373 29071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:59.062345 29071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:59.062495 29071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:59.063022 29071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:59.063791 29071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:01:59.063838 29071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:59.063876 29071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:01:59.063897 29071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:59.072270 29071 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:36765
I20260619 07:01:59.072299 29199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:36765 every 8 connection(s)
I20260619 07:01:59.073210 29071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:01:59.076689 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 29071
I20260619 07:01:59.076829 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 07:01:59.077117 11689 external_mini_cluster.cc:1428] Setting key 43c8e308ba5a5bca8167689ad96fc52c
Jun 19 07:01:59 dist-test-slave-10d0 krb5kdc[28696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852519, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:01:59.086910 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.075001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:42693 (local address 127.11.106.126:39389)
0619 07:01:59.075155 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.075159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.075958 (+ 799us) server_negotiation.cc:407] Connection header received
0619 07:01:59.077556 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.077559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.077645 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.077739 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:59.079067 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.079650 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.080932 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.081145 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.081226 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.083669 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:59.083672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:59.083674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:59.083697 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:59.085136 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:59.085648 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:59.085651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:59.085652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:59.085722 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:59.086019 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:59.086021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:59.086022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:59.086180 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:59.086252 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.086701 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.086793 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 07:01:59.087692 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.075350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:39389 (local address 127.11.106.67:42693)
0619 07:01:59.075800 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:59.075842 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:59.077323 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:59.077890 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:59.077918 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:59.078402 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:59.078904 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.078917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.079762 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.079765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.080823 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.080832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.081270 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.081272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.081405 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.082096 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:01:59.082112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:59.083507 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:59.085235 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:59.085239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:59.085251 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:59.085555 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:59.085798 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:01:59.085801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:01:59.085802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:01:59.085935 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:01:59.086264 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:59.086269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:59.086481 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 07:01:59.086658 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 07:01:59.086894 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260619 07:01:59.088829 29200 heartbeater.cc:347] Connected to a master server at 127.11.106.126:39389
I20260619 07:01:59.089068 29200 heartbeater.cc:464] Registering TS with master...
I20260619 07:01:59.089645 29200 heartbeater.cc:511] Master 127.11.106.126:39389 requested a full tablet report, sending...
I20260619 07:01:59.090844 28740 ts_manager.cc:194] Registered new tserver with Master: 318c85877a8a409cbed3fed81d2dbd59 (127.11.106.67:36765)
I20260619 07:01:59.091424 28740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:42693
I20260619 07:01:59.102535 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:01:59.111507 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.104473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50774 (local address 127.11.106.126:39389)
0619 07:01:59.104623 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.104627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.104774 (+ 147us) server_negotiation.cc:407] Connection header received
0619 07:01:59.104860 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.104862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.104899 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.104964 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:59.105646 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.106141 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.106947 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.107155 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.107232 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.108518 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:59.108520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:59.108522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:01:59.108613 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:59.110046 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:59.110404 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:59.110407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:59.110408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:59.110479 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:01:59.110697 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:01:59.110700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:01:59.110701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:01:59.110901 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:59.111003 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.111280 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.111391 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 07:01:59.114382 28740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:01:59.117321 28740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:01:59.131618 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.125774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.11.106.65:46015)
0619 07:01:59.126248 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.126266 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.126642 (+ 376us) server_negotiation.cc:407] Connection header received
0619 07:01:59.126949 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.126957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.127173 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.127299 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:01:59.127926 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.128775 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.130208 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.130820 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.130945 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.131097 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.131339 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.131452 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":89,"threads_started":1}
I20260619 07:01:59.131713 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.125586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:46015 (local address 127.0.0.1:48178)
0619 07:01:59.126492 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:59.126541 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:59.126755 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:59.127483 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:59.127486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:59.127503 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:01:59.127746 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.127752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.128931 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.128934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.130077 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.130087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.130946 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.130949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.131055 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.131102 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 07:01:59.131277 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 07:01:59.131402 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":133,"thread_start_us":69,"threads_started":1}
I20260619 07:01:59.132091 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.126269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.11.106.67:36765)
0619 07:01:59.126705 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.126719 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.127055 (+ 336us) server_negotiation.cc:407] Connection header received
0619 07:01:59.127135 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.127139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.127280 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.127440 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:01:59.127852 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.128690 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.130278 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.131124 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.131233 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.131397 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.131840 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.131954 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":139,"threads_started":1}
I20260619 07:01:59.132439 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.125773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:36765 (local address 127.0.0.1:54020)
0619 07:01:59.126823 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:59.126837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:59.126937 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:59.127437 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:59.127442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:59.127465 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:01:59.127722 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.127729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.128850 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.128855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.130167 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.130181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.131462 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.131465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.131559 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.131583 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 07:01:59.131656 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 07:01:59.132302 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":120,"spinlock_wait_cycles":2304,"thread_start_us":100,"threads_started":1}
I20260619 07:01:59.132915 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.126295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:45009 (local address 127.0.0.1:55140)
0619 07:01:59.126823 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:59.126837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:59.126937 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:59.127605 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:59.127610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:59.127627 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:01:59.127726 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.127730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.129939 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.129943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.131084 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.131098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.132470 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.132474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.132612 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.132639 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 07:01:59.132712 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 07:01:59.132784 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":1,"spinlock_wait_cycles":7040,"thread_start_us":83,"threads_started":1}
I20260619 07:01:59.133306 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.126659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.11.106.66:45009)
0619 07:01:59.127067 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.127077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.127111 (+ 34us) server_negotiation.cc:407] Connection header received
0619 07:01:59.127179 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.127185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.127349 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.127485 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:01:59.128651 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.129797 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.131222 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.132184 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.132284 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.132406 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.133084 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.133179 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":122,"threads_started":1}
I20260619 07:01:59.135131 28866 tablet_service.cc:1467] Processing CreateTablet for tablet ea7b3fed2959488d9020c37b129bebbe (DEFAULT_TABLE table=test-table [id=20a33a2605784df0ac56a5edf7dd75da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:01:59.135231 29134 tablet_service.cc:1467] Processing CreateTablet for tablet ea7b3fed2959488d9020c37b129bebbe (DEFAULT_TABLE table=test-table [id=20a33a2605784df0ac56a5edf7dd75da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:01:59.135849 29000 tablet_service.cc:1467] Processing CreateTablet for tablet ea7b3fed2959488d9020c37b129bebbe (DEFAULT_TABLE table=test-table [id=20a33a2605784df0ac56a5edf7dd75da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:01:59.136318 28866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7b3fed2959488d9020c37b129bebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:01:59.136318 29134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7b3fed2959488d9020c37b129bebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:01:59.136804 29000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7b3fed2959488d9020c37b129bebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:01:59.142627 29217 tablet_bootstrap.cc:492] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: Bootstrap starting.
I20260619 07:01:59.143987 29218 tablet_bootstrap.cc:492] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: Bootstrap starting.
I20260619 07:01:59.144168 29219 tablet_bootstrap.cc:492] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318: Bootstrap starting.
I20260619 07:01:59.144953 29217 tablet_bootstrap.cc:654] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: Neither blocks nor log segments found. Creating new log.
I20260619 07:01:59.145993 29217 log.cc:824] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: Log is configured to *not* fsync() on all Append() calls
I20260619 07:01:59.146154 29219 tablet_bootstrap.cc:654] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318: Neither blocks nor log segments found. Creating new log.
I20260619 07:01:59.146150 29218 tablet_bootstrap.cc:654] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: Neither blocks nor log segments found. Creating new log.
I20260619 07:01:59.147404 29218 log.cc:824] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: Log is configured to *not* fsync() on all Append() calls
I20260619 07:01:59.147404 29219 log.cc:824] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318: Log is configured to *not* fsync() on all Append() calls
I20260619 07:01:59.148236 29217 tablet_bootstrap.cc:492] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: No bootstrap required, opened a new log
I20260619 07:01:59.148473 29217 ts_tablet_manager.cc:1397] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 07:01:59.149295 29219 tablet_bootstrap.cc:492] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318: No bootstrap required, opened a new log
I20260619 07:01:59.149499 29219 ts_tablet_manager.cc:1397] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 07:01:59.149765 29218 tablet_bootstrap.cc:492] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: No bootstrap required, opened a new log
I20260619 07:01:59.149940 29218 ts_tablet_manager.cc:1397] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 07:01:59.152307 29219 raft_consensus.cc:348] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } }
I20260619 07:01:59.152550 29219 raft_consensus.cc:374] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:01:59.152631 29219 raft_consensus.cc:729] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb15badc63124b8aa5c558d5fb08f318, State: Initialized, Role: FOLLOWER
I20260619 07:01:59.152426 29217 raft_consensus.cc:348] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } }
I20260619 07:01:59.152694 29217 raft_consensus.cc:374] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:01:59.152752 29217 raft_consensus.cc:729] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318c85877a8a409cbed3fed81d2dbd59, State: Initialized, Role: FOLLOWER
I20260619 07:01:59.153133 29219 consensus_queue.cc:260] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } }
I20260619 07:01:59.153195 29217 consensus_queue.cc:260] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } }
I20260619 07:01:59.153712 29218 raft_consensus.cc:348] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } }
I20260619 07:01:59.153940 29218 raft_consensus.cc:374] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:01:59.153990 29218 raft_consensus.cc:729] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a81db2dba0489e9b64f35e2a7bf6a2, State: Initialized, Role: FOLLOWER
I20260619 07:01:59.154114 29219 ts_tablet_manager.cc:1428] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:01:59.154150 29217 ts_tablet_manager.cc:1428] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 07:01:59.154368 29218 consensus_queue.cc:260] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } }
I20260619 07:01:59.154474 29200 heartbeater.cc:503] Master 127.11.106.126:39389 was elected leader, sending a full tablet report...
I20260619 07:01:59.154475 28932 heartbeater.cc:503] Master 127.11.106.126:39389 was elected leader, sending a full tablet report...
I20260619 07:01:59.155213 29218 ts_tablet_manager.cc:1428] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 07:01:59.155541 29066 heartbeater.cc:503] Master 127.11.106.126:39389 was elected leader, sending a full tablet report...
W20260619 07:01:59.324257 29201 tablet.cc:2369] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:01:59.330734 29224 raft_consensus.cc:482] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:01:59.330915 29224 raft_consensus.cc:504] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } }
I20260619 07:01:59.332141 29224 leader_election.cc:288] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb15badc63124b8aa5c558d5fb08f318 (127.11.106.65:46015), 64a81db2dba0489e9b64f35e2a7bf6a2 (127.11.106.66:45009)
I20260619 07:01:59.336921 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.332470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:33339 (local address 127.11.106.65:46015)
0619 07:01:59.332652 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.332657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.332677 (+ 20us) server_negotiation.cc:407] Connection header received
0619 07:01:59.332840 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.332842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.332914 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.332993 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:01:59.333318 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.334116 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.335623 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.336270 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.336390 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.336443 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.336708 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.336792 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 07:01:59.336930 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.332356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:46015 (local address 127.11.106.67:33339)
0619 07:01:59.332511 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:59.332528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:59.332694 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:59.333027 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:59.333030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:59.333063 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:01:59.333203 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.333211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.334222 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.334225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.335476 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.335487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.336403 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.336407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.336512 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.336588 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 07:01:59.336702 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 07:01:59.336788 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 07:01:59.337018 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.332817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:41935 (local address 127.11.106.66:45009)
0619 07:01:59.332966 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.332973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.333189 (+ 216us) server_negotiation.cc:407] Connection header received
0619 07:01:59.333286 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.333289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.333342 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.333424 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:01:59.333655 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.334341 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.335690 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.336270 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.336411 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.336455 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.336800 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.336875 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 07:01:59.336961 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.332686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:45009 (local address 127.11.106.67:41935)
0619 07:01:59.333082 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:59.333095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:59.333189 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:59.333437 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:59.333439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:59.333454 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:01:59.333546 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.333550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.334433 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.334436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.335596 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.335607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.336518 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.336523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.336668 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.336695 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 07:01:59.336766 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 07:01:59.336829 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":142,"threads_started":1}
I20260619 07:01:59.337667 28886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7b3fed2959488d9020c37b129bebbe" candidate_uuid: "318c85877a8a409cbed3fed81d2dbd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb15badc63124b8aa5c558d5fb08f318" is_pre_election: true
I20260619 07:01:59.337980 28886 raft_consensus.cc:2454] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 318c85877a8a409cbed3fed81d2dbd59 in term 0.
I20260619 07:01:59.337953 29020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7b3fed2959488d9020c37b129bebbe" candidate_uuid: "318c85877a8a409cbed3fed81d2dbd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" is_pre_election: true
I20260619 07:01:59.338243 29020 raft_consensus.cc:2454] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 318c85877a8a409cbed3fed81d2dbd59 in term 0.
I20260619 07:01:59.338472 29087 leader_election.cc:302] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318c85877a8a409cbed3fed81d2dbd59, fb15badc63124b8aa5c558d5fb08f318; no voters:
I20260619 07:01:59.338759 29224 raft_consensus.cc:2781] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:01:59.338832 29224 raft_consensus.cc:482] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:01:59.338863 29224 raft_consensus.cc:3037] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:01:59.339888 29224 raft_consensus.cc:504] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } }
I20260619 07:01:59.340224 29224 leader_election.cc:288] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [CANDIDATE]: Term 1 election: Requested vote from peers fb15badc63124b8aa5c558d5fb08f318 (127.11.106.65:46015), 64a81db2dba0489e9b64f35e2a7bf6a2 (127.11.106.66:45009)
I20260619 07:01:59.340641 28886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7b3fed2959488d9020c37b129bebbe" candidate_uuid: "318c85877a8a409cbed3fed81d2dbd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb15badc63124b8aa5c558d5fb08f318"
I20260619 07:01:59.340674 29020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7b3fed2959488d9020c37b129bebbe" candidate_uuid: "318c85877a8a409cbed3fed81d2dbd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2"
I20260619 07:01:59.340776 28886 raft_consensus.cc:3037] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:01:59.340792 29020 raft_consensus.cc:3037] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:01:59.341775 29020 raft_consensus.cc:2454] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 318c85877a8a409cbed3fed81d2dbd59 in term 1.
I20260619 07:01:59.341877 28886 raft_consensus.cc:2454] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 318c85877a8a409cbed3fed81d2dbd59 in term 1.
I20260619 07:01:59.342060 29087 leader_election.cc:302] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318c85877a8a409cbed3fed81d2dbd59, 64a81db2dba0489e9b64f35e2a7bf6a2; no voters:
I20260619 07:01:59.342234 29224 raft_consensus.cc:2781] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:01:59.342435 29224 raft_consensus.cc:686] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 1 LEADER]: Becoming Leader. State: Replica: 318c85877a8a409cbed3fed81d2dbd59, State: Running, Role: LEADER
I20260619 07:01:59.342751 29224 consensus_queue.cc:237] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } }
I20260619 07:01:59.345827 28740 catalog_manager.cc:5495] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 318c85877a8a409cbed3fed81d2dbd59 (127.11.106.67). New cstate: current_term: 1 leader_uuid: "318c85877a8a409cbed3fed81d2dbd59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "318c85877a8a409cbed3fed81d2dbd59" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 36765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 } health_report { overall_health: UNKNOWN } } }
W20260619 07:01:59.346565 29067 tablet.cc:2369] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:01:59.366859 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.362848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.11.106.67:36765)
0619 07:01:59.362961 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.362970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.362984 (+ 14us) server_negotiation.cc:407] Connection header received
0619 07:01:59.363097 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.363099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.363153 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.363242 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:01:59.363526 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.364012 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.365249 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.365511 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.365597 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.365989 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:01:59.366369 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:01:59.366468 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.366598 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.366650 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 07:01:59.372567 29020 raft_consensus.cc:1261] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [term 1 FOLLOWER]: Refusing update from remote peer 318c85877a8a409cbed3fed81d2dbd59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:01:59.372567 28886 raft_consensus.cc:1261] T ea7b3fed2959488d9020c37b129bebbe P fb15badc63124b8aa5c558d5fb08f318 [term 1 FOLLOWER]: Refusing update from remote peer 318c85877a8a409cbed3fed81d2dbd59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:01:59.373281 29224 consensus_queue.cc:1035] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64a81db2dba0489e9b64f35e2a7bf6a2" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:01:59.373458 29227 consensus_queue.cc:1035] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb15badc63124b8aa5c558d5fb08f318" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:01:59.381319 29232 mvcc.cc:205] Tried to move back new op lower bound from 7298467919342346240 to 7298467919230296064. Current Snapshot: MvccSnapshot[applied={T|T < 7298467919342346240}]
I20260619 07:01:59.391031 28740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:01:59.391211 28740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:01:59.393612 28740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e0aba092f5948619e5b615db21e0997: Sending DeleteTablet for 3 replicas of tablet ea7b3fed2959488d9020c37b129bebbe
I20260619 07:01:59.394347 29134 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7b3fed2959488d9020c37b129bebbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:59 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:54020
I20260619 07:01:59.394393 29000 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7b3fed2959488d9020c37b129bebbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:59 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:55140
I20260619 07:01:59.394419 28866 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7b3fed2959488d9020c37b129bebbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:01:59 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:48178
I20260619 07:01:59.394797 29241 tablet_replica.cc:330] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: stopping tablet replica
I20260619 07:01:59.395114 29241 raft_consensus.cc:2229] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:01:59.395201 29242 tablet_replica.cc:330] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: stopping tablet replica
I20260619 07:01:59.395429 29241 raft_consensus.cc:2258] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:01:59.395458 29242 raft_consensus.cc:2229] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 1 LEADER]: Raft consensus shutting down.
I20260619 07:01:59.395615 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 28803
I20260619 07:01:59.395805 29242 raft_consensus.cc:2258] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:01:59.397953 29242 ts_tablet_manager.cc:1905] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:01:59.397953 29241 ts_tablet_manager.cc:1905] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:01:59.400707 29241 ts_tablet_manager.cc:1918] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:01:59.400707 29242 ts_tablet_manager.cc:1918] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:01:59.400807 29242 log.cc:1196] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/ea7b3fed2959488d9020c37b129bebbe
I20260619 07:01:59.400807 29241 log.cc:1196] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/ea7b3fed2959488d9020c37b129bebbe
I20260619 07:01:59.401145 29242 ts_tablet_manager.cc:1939] T ea7b3fed2959488d9020c37b129bebbe P 318c85877a8a409cbed3fed81d2dbd59: Deleting consensus metadata
I20260619 07:01:59.401165 29241 ts_tablet_manager.cc:1939] T ea7b3fed2959488d9020c37b129bebbe P 64a81db2dba0489e9b64f35e2a7bf6a2: Deleting consensus metadata
I20260619 07:01:59.402340 28725 catalog_manager.cc:4841] TS 64a81db2dba0489e9b64f35e2a7bf6a2 (127.11.106.66:45009): tablet ea7b3fed2959488d9020c37b129bebbe (table test-table [id=20a33a2605784df0ac56a5edf7dd75da]) successfully deleted
I20260619 07:01:59.402565 28727 catalog_manager.cc:4841] TS 318c85877a8a409cbed3fed81d2dbd59 (127.11.106.67:36765): tablet ea7b3fed2959488d9020c37b129bebbe (table test-table [id=20a33a2605784df0ac56a5edf7dd75da]) successfully deleted
W20260619 07:01:59.403708 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:46015 (error 108)
I20260619 07:01:59.404563 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 28937
I20260619 07:01:59.404773 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.404467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:46015 (local address 127.0.0.1:48184)
0619 07:01:59.404626 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:59.404695 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:46015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 07:01:59.409163 28725 catalog_manager.cc:4584] TS fb15badc63124b8aa5c558d5fb08f318 (127.11.106.65:46015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea7b3fed2959488d9020c37b129bebbe: Network error: Client connection negotiation failed: client connection to 127.11.106.65:46015: connect: Connection refused (error 111)
I20260619 07:01:59.411808 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 29071
I20260619 07:01:59.417765 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 28709
2026-06-19T07:01:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2454 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T07:01:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:01:59Z Disabled control of system clock
I20260619 07:01:59.447585 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:44345
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:44345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 07:01:59.557818 29248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:59.558104 29248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:59.558158 29248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:59.562757 29248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:01:59.562817 29248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:59.562834 29248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:59.562851 29248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:01:59.562866 29248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:01:59.567668 29248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:44345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:44345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:59.568809 29248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:59.570223 29248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:59.578866 29253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:59.578852 29254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:59.578852 29256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:59.579077 29248 server_base.cc:1034] running on GCE node
I20260619 07:01:59.579711 29248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:59.580907 29248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:59.582084 29248 hybrid_clock.cc:648] HybridClock initialized: now 1781852519582054 us; error 41 us; skew 500 ppm
I20260619 07:01:59.584024 29248 webserver.cc:460] Webserver started at http://127.11.106.126:41565/ using document root <none> and password file <none>
I20260619 07:01:59.584700 29248 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:59.584769 29248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:59.585007 29248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:01:59.587049 29248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "db6bd5d164ed40da88967a810c6c3222"
format_stamp: "Formatted at 2026-06-19 07:01:59 on dist-test-slave-10d0"
server_key: "b916f3fff51714cfd768291a4a800b76"
server_key_iv: "1436199eb8c934b9fd993c7f8539a483"
server_key_version: "encryptionkey@0"
I20260619 07:01:59.587625 29248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "db6bd5d164ed40da88967a810c6c3222"
format_stamp: "Formatted at 2026-06-19 07:01:59 on dist-test-slave-10d0"
server_key: "b916f3fff51714cfd768291a4a800b76"
server_key_iv: "1436199eb8c934b9fd993c7f8539a483"
server_key_version: "encryptionkey@0"
I20260619 07:01:59.591257 29248 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 07:01:59.593714 29262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:59.594966 29248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:01:59.595113 29248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "db6bd5d164ed40da88967a810c6c3222"
format_stamp: "Formatted at 2026-06-19 07:01:59 on dist-test-slave-10d0"
server_key: "b916f3fff51714cfd768291a4a800b76"
server_key_iv: "1436199eb8c934b9fd993c7f8539a483"
server_key_version: "encryptionkey@0"
I20260619 07:01:59.595228 29248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:59.612423 29248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:59.613193 29248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:59.613399 29248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:59.620488 29248 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:44345
I20260619 07:01:59.620496 29314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:44345 every 8 connection(s)
I20260619 07:01:59.621531 29248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:01:59.623260 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 29248
I20260619 07:01:59.623421 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:01:59.623778 11689 external_mini_cluster.cc:1428] Setting key 933cd9d5df3d3ee5fd42033060aa215c
I20260619 07:01:59.625123 29315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:01:59.631531 29315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222: Bootstrap starting.
I20260619 07:01:59.633368 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.625419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.11.106.126:44345)
0619 07:01:59.626118 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.626133 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.626167 (+ 34us) server_negotiation.cc:407] Connection header received
0619 07:01:59.626798 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.626813 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.627201 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.627477 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:59.627796 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.628828 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.630000 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.630445 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.630562 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.631434 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:59.631439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:59.631453 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:01:59.631908 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:59.631963 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:01:59.632000 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:59.632076 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.632246 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.632588 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":161,"threads_started":1}
I20260619 07:01:59.634055 29315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222: Neither blocks nor log segments found. Creating new log.
I20260619 07:01:59.634925 29315 log.cc:824] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222: Log is configured to *not* fsync() on all Append() calls
I20260619 07:01:59.636955 29315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222: No bootstrap required, opened a new log
I20260619 07:01:59.640348 29315 raft_consensus.cc:348] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6bd5d164ed40da88967a810c6c3222" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 44345 } }
I20260619 07:01:59.640619 29315 raft_consensus.cc:374] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:01:59.640668 29315 raft_consensus.cc:729] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6bd5d164ed40da88967a810c6c3222, State: Initialized, Role: FOLLOWER
I20260619 07:01:59.641074 29315 consensus_queue.cc:260] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6bd5d164ed40da88967a810c6c3222" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 44345 } }
I20260619 07:01:59.641220 29315 raft_consensus.cc:388] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:01:59.641286 29315 raft_consensus.cc:482] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:01:59.641379 29315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:01:59.642472 29315 raft_consensus.cc:504] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6bd5d164ed40da88967a810c6c3222" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 44345 } }
I20260619 07:01:59.642789 29315 leader_election.cc:302] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db6bd5d164ed40da88967a810c6c3222; no voters:
I20260619 07:01:59.643034 29315 leader_election.cc:288] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:01:59.643177 29320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:01:59.643433 29320 raft_consensus.cc:686] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [term 1 LEADER]: Becoming Leader. State: Replica: db6bd5d164ed40da88967a810c6c3222, State: Running, Role: LEADER
I20260619 07:01:59.643749 29320 consensus_queue.cc:237] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6bd5d164ed40da88967a810c6c3222" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 44345 } }
I20260619 07:01:59.644256 29315 sys_catalog.cc:564] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:01:59.645525 29322 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db6bd5d164ed40da88967a810c6c3222. Latest consensus state: current_term: 1 leader_uuid: "db6bd5d164ed40da88967a810c6c3222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6bd5d164ed40da88967a810c6c3222" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 44345 } } }
I20260619 07:01:59.645526 29321 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db6bd5d164ed40da88967a810c6c3222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6bd5d164ed40da88967a810c6c3222" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 44345 } } }
I20260619 07:01:59.645676 29322 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [sys.catalog]: This master's current role is: LEADER
I20260619 07:01:59.645676 29321 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222 [sys.catalog]: This master's current role is: LEADER
I20260619 07:01:59.646198 29329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:01:59.649158 29329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:01:59.654582 29329 catalog_manager.cc:1347] Generated new cluster ID: a99676b1300b45b7b0e7cb49e44ea068
I20260619 07:01:59.654644 29329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:01:59.672469 29329 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:01:59.673746 29329 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:01:59.684810 29329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222: Generated new TSK 0
I20260619 07:01:59.685657 29329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:01:59.688380 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:44345
--builtin_ntp_servers=127.11.106.84:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 07:01:59.800984 29339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:59.801261 29339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:59.801329 29339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:01:59.805289 29339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:01:59.805356 29339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:01:59.805442 29339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:01:59.810029 29339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:44345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:01:59.811028 29339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:01:59.812407 29339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:01:59.821643 29345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:59.821585 29347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:01:59.821585 29344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:01:59.821978 29339 server_base.cc:1034] running on GCE node
I20260619 07:01:59.822376 29339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:01:59.823043 29339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:01:59.824231 29339 hybrid_clock.cc:648] HybridClock initialized: now 1781852519824198 us; error 47 us; skew 500 ppm
I20260619 07:01:59.825949 29339 webserver.cc:460] Webserver started at http://127.11.106.65:34419/ using document root <none> and password file <none>
I20260619 07:01:59.826512 29339 fs_manager.cc:359] Metadata directory not provided
I20260619 07:01:59.826603 29339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:01:59.826830 29339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:01:59.828855 29339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "4af77b80e21c4f85a8c12991eb8e154d"
format_stamp: "Formatted at 2026-06-19 07:01:59 on dist-test-slave-10d0"
server_key: "1fe6809bd404aab66471a246c86d0d22"
server_key_iv: "3e4d22fb9f859f2523fbb95c809b7e8a"
server_key_version: "encryptionkey@0"
I20260619 07:01:59.829403 29339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "4af77b80e21c4f85a8c12991eb8e154d"
format_stamp: "Formatted at 2026-06-19 07:01:59 on dist-test-slave-10d0"
server_key: "1fe6809bd404aab66471a246c86d0d22"
server_key_iv: "3e4d22fb9f859f2523fbb95c809b7e8a"
server_key_version: "encryptionkey@0"
I20260619 07:01:59.833029 29339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 07:01:59.835454 29353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:59.836660 29339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:01:59.836787 29339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "4af77b80e21c4f85a8c12991eb8e154d"
format_stamp: "Formatted at 2026-06-19 07:01:59 on dist-test-slave-10d0"
server_key: "1fe6809bd404aab66471a246c86d0d22"
server_key_iv: "3e4d22fb9f859f2523fbb95c809b7e8a"
server_key_version: "encryptionkey@0"
I20260619 07:01:59.836894 29339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:01:59.862680 29339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:01:59.863394 29339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:01:59.863605 29339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:01:59.864204 29339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:01:59.865338 29339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:01:59.865401 29339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:59.865458 29339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:01:59.865521 29339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:01:59.874440 29339 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:35343
I20260619 07:01:59.874470 29466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:35343 every 8 connection(s)
I20260619 07:01:59.875383 29339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:01:59.884188 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.877311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:36823 (local address 127.11.106.126:44345)
0619 07:01:59.877448 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 07:01:59.877452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:01:59.878462 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 07:01:59.879820 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:01:59.879823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:01:59.879869 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:01:59.879945 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:01:59.880886 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.881403 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.882467 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.882678 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.882754 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.883767 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:01:59.883770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:01:59.883772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:01:59.883785 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:01:59.883802 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:01:59.883810 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:01:59.883867 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 07:01:59.884009 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 07:01:59.884075 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 07:01:59.884958 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 29339
I20260619 07:01:59.885109 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 07:01:59.885278 29469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:01:59.877700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:44345 (local address 127.11.106.65:36823)
0619 07:01:59.878330 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 07:01:59.878376 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 07:01:59.879614 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:01:59.880116 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:01:59.880122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:01:59.880206 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:01:59.880767 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.880782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.881509 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.881512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.882362 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:01:59.882371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:01:59.882814 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:01:59.882816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:01:59.882951 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:01:59.883521 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 07:01:59.883522 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:01:59.883568 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 07:01:59.883628 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:01:59.883883 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:01:59.883889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:01:59.883894 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 07:01:59.883986 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 07:01:59.884192 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260619 07:01:59.885440 11689 external_mini_cluster.cc:1428] Setting key 35ccaab1fe2e809c4e5b886ce2472708
I20260619 07:01:59.886540 29467 heartbeater.cc:347] Connected to a master server at 127.11.106.126:44345
I20260619 07:01:59.886771 29467 heartbeater.cc:464] Registering TS with master...
I20260619 07:01:59.887279 29467 heartbeater.cc:511] Master 127.11.106.126:44345 requested a full tablet report, sending...
I20260619 07:01:59.887295 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:44345
--builtin_ntp_servers=127.11.106.84:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 07:01:59.888796 29279 ts_manager.cc:194] Registered new tserver with Master: 4af77b80e21c4f85a8c12991eb8e154d (127.11.106.65:35343)
I20260619 07:01:59.890040 29279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.65:36823
W20260619 07:01:59.999126 29470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:01:59.999410 29470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:01:59.999449 29470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:00.003655 29470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:00.003712 29470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:00.003768 29470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:02:00.008625 29470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:44345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:00.009819 29470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:00.011234 29470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:00.021832 29478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:00.021855 29475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:00.021826 29476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:00.022224 29470 server_base.cc:1034] running on GCE node
I20260619 07:02:00.022643 29470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:00.023346 29470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:00.024528 29470 hybrid_clock.cc:648] HybridClock initialized: now 1781852520024510 us; error 30 us; skew 500 ppm
I20260619 07:02:00.026516 29470 webserver.cc:460] Webserver started at http://127.11.106.66:34603/ using document root <none> and password file <none>
I20260619 07:02:00.027127 29470 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:00.027189 29470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:00.027446 29470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:00.029646 29470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "e9ebefc474324be5935325d92a2a7dac"
format_stamp: "Formatted at 2026-06-19 07:02:00 on dist-test-slave-10d0"
server_key: "ddf607db57f9355b723e49c98256cc90"
server_key_iv: "c3d3d946a72284401629c75eff05bcd4"
server_key_version: "encryptionkey@0"
I20260619 07:02:00.030226 29470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "e9ebefc474324be5935325d92a2a7dac"
format_stamp: "Formatted at 2026-06-19 07:02:00 on dist-test-slave-10d0"
server_key: "ddf607db57f9355b723e49c98256cc90"
server_key_iv: "c3d3d946a72284401629c75eff05bcd4"
server_key_version: "encryptionkey@0"
I20260619 07:02:00.034140 29470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 07:02:00.036749 29484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:00.038044 29470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 07:02:00.038172 29470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "e9ebefc474324be5935325d92a2a7dac"
format_stamp: "Formatted at 2026-06-19 07:02:00 on dist-test-slave-10d0"
server_key: "ddf607db57f9355b723e49c98256cc90"
server_key_iv: "c3d3d946a72284401629c75eff05bcd4"
server_key_version: "encryptionkey@0"
I20260619 07:02:00.038317 29470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:00.054950 29470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:00.055725 29470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:00.055943 29470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:00.056663 29470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:00.057689 29470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:00.057744 29470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:00.057813 29470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:00.057847 29470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:00.067049 29470 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:37357
I20260619 07:02:00.067075 29597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:37357 every 8 connection(s)
I20260619 07:02:00.067986 29470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:02:00.074429 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 29470
I20260619 07:02:00.074579 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 07:02:00.074865 11689 external_mini_cluster.cc:1428] Setting key f7dc2df17dd31f71581463e3a87ce6ba
I20260619 07:02:00.076918 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:44345
--builtin_ntp_servers=127.11.106.84:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 07:02:00.077654 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.069874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:59255 (local address 127.11.106.126:44345)
0619 07:02:00.070053 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:00.070058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:00.071167 (+ 1109us) server_negotiation.cc:407] Connection header received
0619 07:02:00.072524 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:00.072527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:00.072632 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:00.072709 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:00.073601 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.074120 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.075476 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.075749 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.075874 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.077170 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:00.077174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:00.077177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:00.077197 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:00.077214 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:00.077221 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:00.077256 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:00.077449 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 07:02:00.077508 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":73}
I20260619 07:02:00.078531 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.070309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:44345 (local address 127.11.106.66:59255)
0619 07:02:00.070993 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:00.071036 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:00.072318 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:00.072882 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:00.072889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:00.072976 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:00.073473 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.073485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.074255 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.074257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.075314 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.075324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.075914 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.075918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.076116 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.076858 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 07:02:00.076859 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:00.076900 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 07:02:00.076963 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:00.077309 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:00.077317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:00.077321 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 07:02:00.077446 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 07:02:00.077740 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
I20260619 07:02:00.079730 29598 heartbeater.cc:347] Connected to a master server at 127.11.106.126:44345
I20260619 07:02:00.079977 29598 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:00.080636 29598 heartbeater.cc:511] Master 127.11.106.126:44345 requested a full tablet report, sending...
I20260619 07:02:00.081822 29279 ts_manager.cc:194] Registered new tserver with Master: e9ebefc474324be5935325d92a2a7dac (127.11.106.66:37357)
I20260619 07:02:00.082392 29279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.66:59255
W20260619 07:02:00.188800 29601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:00.189073 29601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:00.189109 29601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:00.193135 29601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:00.193185 29601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:00.193271 29601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:02:00.197950 29601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:44345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:00.199023 29601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:00.200398 29601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:00.209863 29609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:00.209904 29606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:00.209851 29607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:00.212638 29601 server_base.cc:1034] running on GCE node
I20260619 07:02:00.213182 29601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:00.213917 29601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:00.215103 29601 hybrid_clock.cc:648] HybridClock initialized: now 1781852520215051 us; error 43 us; skew 500 ppm
I20260619 07:02:00.217397 29601 webserver.cc:460] Webserver started at http://127.11.106.67:43649/ using document root <none> and password file <none>
I20260619 07:02:00.218070 29601 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:00.218173 29601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:00.218443 29601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:00.220578 29601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "21bd90e650de470e9fdf3932416086cb"
format_stamp: "Formatted at 2026-06-19 07:02:00 on dist-test-slave-10d0"
server_key: "68126a3359ad987158f4053b56cef020"
server_key_iv: "77ef4dd5bb50dd201ea5d3109cb1f669"
server_key_version: "encryptionkey@0"
I20260619 07:02:00.221156 29601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "21bd90e650de470e9fdf3932416086cb"
format_stamp: "Formatted at 2026-06-19 07:02:00 on dist-test-slave-10d0"
server_key: "68126a3359ad987158f4053b56cef020"
server_key_iv: "77ef4dd5bb50dd201ea5d3109cb1f669"
server_key_version: "encryptionkey@0"
I20260619 07:02:00.224920 29601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:02:00.227294 29615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:00.228472 29601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:00.228647 29601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "21bd90e650de470e9fdf3932416086cb"
format_stamp: "Formatted at 2026-06-19 07:02:00 on dist-test-slave-10d0"
server_key: "68126a3359ad987158f4053b56cef020"
server_key_iv: "77ef4dd5bb50dd201ea5d3109cb1f669"
server_key_version: "encryptionkey@0"
I20260619 07:02:00.228754 29601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:00.252951 29601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:00.253700 29601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:00.253896 29601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:00.254535 29601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:00.255462 29601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:00.255507 29601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:00.255576 29601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:00.255622 29601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:00.266347 29601 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:38937
I20260619 07:02:00.266419 29728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:38937 every 8 connection(s)
I20260619 07:02:00.267580 29601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:02:00.274050 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 29601
I20260619 07:02:00.274169 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 07:02:00.274478 11689 external_mini_cluster.cc:1428] Setting key 423840197387b25b72de2f117ce4da0a
I20260619 07:02:00.280754 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.270302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:47723 (local address 127.11.106.126:44345)
0619 07:02:00.270458 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:00.270463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:00.272058 (+ 1595us) server_negotiation.cc:407] Connection header received
0619 07:02:00.274211 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:00.274219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:00.274290 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:00.274358 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:00.275938 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.276962 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.278526 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.278817 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.278936 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.280214 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:00.280217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:00.280220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:00.280234 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:00.280257 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:00.280286 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:00.280350 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:00.280519 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 07:02:00.280607 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 07:02:00.281622 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.270888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:44345 (local address 127.11.106.67:47723)
0619 07:02:00.271876 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:00.271942 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:00.273894 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:00.274841 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:00.274853 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:00.274975 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:00.275710 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.275736 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.277081 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.277085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.278415 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.278428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.278975 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.278980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.279178 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.279899 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 07:02:00.279901 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:00.279966 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 07:02:00.280057 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:00.280363 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:00.280372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:00.280378 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 07:02:00.280520 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 07:02:00.280845 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":133,"threads_started":1}
I20260619 07:02:00.282856 29729 heartbeater.cc:347] Connected to a master server at 127.11.106.126:44345
I20260619 07:02:00.283134 29729 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:00.283815 29729 heartbeater.cc:511] Master 127.11.106.126:44345 requested a full tablet report, sending...
I20260619 07:02:00.285193 29279 ts_manager.cc:194] Registered new tserver with Master: 21bd90e650de470e9fdf3932416086cb (127.11.106.67:38937)
I20260619 07:02:00.286034 29279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.67:47723
I20260619 07:02:00.289821 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:02:00.297302 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.292166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.11.106.126:44345)
0619 07:02:00.292305 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:00.292310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:00.292493 (+ 183us) server_negotiation.cc:407] Connection header received
0619 07:02:00.292662 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:00.292666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:00.292728 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:00.292813 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:00.293156 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.294037 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.295427 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.295814 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.295955 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.296880 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:00.296883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:00.296885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:00.296898 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:00.296915 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:00.296923 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:00.296983 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:00.297114 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 07:02:00.297160 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 07:02:00.300604 29279 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:02:00.305999 29279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:02:00.320819 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.315052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:35343 (local address 127.0.0.1:56674)
0619 07:02:00.315455 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:00.315511 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:00.315655 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:00.316194 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:00.316200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:00.316224 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:00.316359 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.316366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.317763 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.317766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.319103 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.319112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.320200 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.320203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.320333 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.320374 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 07:02:00.320486 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 07:02:00.320677 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":28,"thread_start_us":110,"threads_started":1}
I20260619 07:02:00.320823 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.315241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.11.106.65:35343)
0619 07:02:00.315604 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:00.315618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:00.315644 (+ 26us) server_negotiation.cc:407] Connection header received
0619 07:02:00.315799 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:00.315806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:00.316020 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:00.316205 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:00.316487 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.317658 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.319219 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.320078 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.320189 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.320402 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:00.320512 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 07:02:00.320648 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
I20260619 07:02:00.321645 29744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.315850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:38937 (local address 127.0.0.1:38630)
0619 07:02:00.316694 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:00.316708 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:00.316789 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:00.317448 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:00.317450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:00.317467 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:00.317566 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.317571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.319004 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.319008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.320181 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.320191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.321246 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.321250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.321348 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.321375 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 07:02:00.321438 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 07:02:00.321501 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260619 07:02:00.321746 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.316002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.11.106.67:38937)
0619 07:02:00.317019 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:00.317030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:00.317052 (+ 22us) server_negotiation.cc:407] Connection header received
0619 07:02:00.317120 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:00.317125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:00.317304 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:00.317444 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:00.317691 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.318905 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.320322 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.321139 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.321244 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.321419 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:00.321484 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 07:02:00.321575 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":164,"threads_started":1}
I20260619 07:02:00.322273 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.316192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:37357 (local address 127.0.0.1:58632)
0619 07:02:00.317665 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:00.317678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:00.317766 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:00.318221 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:00.318224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:00.318241 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:00.318329 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.318336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.319496 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.319499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.320676 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.320685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.321901 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.321904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.321990 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.322011 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 07:02:00.322083 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 07:02:00.322149 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1408,"mutex_wait_us":30,"thread_start_us":77,"threads_started":1}
I20260619 07:02:00.322763 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.316768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.11.106.66:37357)
0619 07:02:00.317084 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:00.317098 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:00.317867 (+ 769us) server_negotiation.cc:407] Connection header received
0619 07:02:00.317937 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:00.317940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:00.318073 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:00.318248 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:00.318432 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.319368 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.320807 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.321546 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.321657 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.321832 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:00.322497 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 07:02:00.322603 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
I20260619 07:02:00.324249 29401 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4a7040df3c4edbb9facb771a330606 (DEFAULT_TABLE table=test-table [id=fd6089a058914a409223738c68fc93da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:00.324263 29663 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4a7040df3c4edbb9facb771a330606 (DEFAULT_TABLE table=test-table [id=fd6089a058914a409223738c68fc93da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:00.325371 29663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4a7040df3c4edbb9facb771a330606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:00.325654 29401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4a7040df3c4edbb9facb771a330606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:00.326378 29532 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4a7040df3c4edbb9facb771a330606 (DEFAULT_TABLE table=test-table [id=fd6089a058914a409223738c68fc93da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:00.327306 29532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4a7040df3c4edbb9facb771a330606. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:00.332649 29747 tablet_bootstrap.cc:492] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: Bootstrap starting.
I20260619 07:02:00.333096 29746 tablet_bootstrap.cc:492] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d: Bootstrap starting.
I20260619 07:02:00.333886 29748 tablet_bootstrap.cc:492] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: Bootstrap starting.
I20260619 07:02:00.335462 29746 tablet_bootstrap.cc:654] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:00.336041 29748 tablet_bootstrap.cc:654] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:00.336041 29747 tablet_bootstrap.cc:654] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:00.336304 29746 log.cc:824] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:00.336911 29747 log.cc:824] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:00.336911 29748 log.cc:824] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:00.338472 29746 tablet_bootstrap.cc:492] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d: No bootstrap required, opened a new log
I20260619 07:02:00.338693 29746 ts_tablet_manager.cc:1397] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 07:02:00.338766 29748 tablet_bootstrap.cc:492] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: No bootstrap required, opened a new log
I20260619 07:02:00.338766 29747 tablet_bootstrap.cc:492] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: No bootstrap required, opened a new log
I20260619 07:02:00.338933 29748 ts_tablet_manager.cc:1397] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 07:02:00.338933 29747 ts_tablet_manager.cc:1397] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 07:02:00.342839 29746 raft_consensus.cc:348] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } }
I20260619 07:02:00.343092 29746 raft_consensus.cc:374] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:00.343148 29746 raft_consensus.cc:729] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af77b80e21c4f85a8c12991eb8e154d, State: Initialized, Role: FOLLOWER
I20260619 07:02:00.343108 29748 raft_consensus.cc:348] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } }
I20260619 07:02:00.343101 29747 raft_consensus.cc:348] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } }
I20260619 07:02:00.343338 29748 raft_consensus.cc:374] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:00.343338 29747 raft_consensus.cc:374] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:00.343393 29747 raft_consensus.cc:729] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bd90e650de470e9fdf3932416086cb, State: Initialized, Role: FOLLOWER
I20260619 07:02:00.343393 29748 raft_consensus.cc:729] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ebefc474324be5935325d92a2a7dac, State: Initialized, Role: FOLLOWER
I20260619 07:02:00.343732 29746 consensus_queue.cc:260] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } }
I20260619 07:02:00.343809 29747 consensus_queue.cc:260] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } }
I20260619 07:02:00.343848 29748 consensus_queue.cc:260] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } }
I20260619 07:02:00.344825 29747 ts_tablet_manager.cc:1428] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260619 07:02:00.344857 29748 ts_tablet_manager.cc:1428] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260619 07:02:00.344966 29746 ts_tablet_manager.cc:1428] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 07:02:00.345175 29598 heartbeater.cc:503] Master 127.11.106.126:44345 was elected leader, sending a full tablet report...
I20260619 07:02:00.345175 29729 heartbeater.cc:503] Master 127.11.106.126:44345 was elected leader, sending a full tablet report...
I20260619 07:02:00.345310 29467 heartbeater.cc:503] Master 127.11.106.126:44345 was elected leader, sending a full tablet report...
W20260619 07:02:00.376689 29468 tablet.cc:2369] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:02:00.422154 29753 raft_consensus.cc:482] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:02:00.422371 29753 raft_consensus.cc:504] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } }
I20260619 07:02:00.423557 29753 leader_election.cc:288] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af77b80e21c4f85a8c12991eb8e154d (127.11.106.65:35343), 21bd90e650de470e9fdf3932416086cb (127.11.106.67:38937)
I20260619 07:02:00.427279 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.423811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:35343 (local address 127.11.106.66:54497)
0619 07:02:00.423970 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:00.423988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:00.424106 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:00.424413 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:00.424416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:00.424436 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:00.424620 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.424628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.425353 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.425359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.426138 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.426147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.426835 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.426838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.426937 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.426983 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 07:02:00.427073 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 07:02:00.427146 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 07:02:00.427278 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.423969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:54497 (local address 127.11.106.65:35343)
0619 07:02:00.424130 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:00.424135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:00.424155 (+ 20us) server_negotiation.cc:407] Connection header received
0619 07:02:00.424218 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:00.424221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:00.424282 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:00.424365 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:00.424745 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.425242 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.426267 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.426716 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.426834 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.426887 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:00.427100 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 07:02:00.427169 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 07:02:00.427745 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.424261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:48581 (local address 127.11.106.67:38937)
0619 07:02:00.424404 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:00.424408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:00.424532 (+ 124us) server_negotiation.cc:407] Connection header received
0619 07:02:00.424717 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:00.424720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:00.424768 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:00.424852 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:00.425115 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.425592 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.426571 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.427002 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.427143 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.427188 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:00.427533 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 07:02:00.427619 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 07:02:00.427929 29421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4a7040df3c4edbb9facb771a330606" candidate_uuid: "e9ebefc474324be5935325d92a2a7dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af77b80e21c4f85a8c12991eb8e154d" is_pre_election: true
I20260619 07:02:00.428094 29755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.424128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:38937 (local address 127.11.106.66:48581)
0619 07:02:00.424432 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:00.424446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:00.424587 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:00.424863 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:00.424866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:00.424882 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:00.425005 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.425010 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.425703 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.425707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.426447 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:00.426455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.427144 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.427147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:00.427245 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.427268 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 07:02:00.427897 (+ 629us) client_negotiation.cc:240] Negotiation successful
0619 07:02:00.427967 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":114,"threads_started":1}
I20260619 07:02:00.428268 29421 raft_consensus.cc:2454] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9ebefc474324be5935325d92a2a7dac in term 0.
I20260619 07:02:00.428786 29488 leader_election.cc:302] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af77b80e21c4f85a8c12991eb8e154d, e9ebefc474324be5935325d92a2a7dac; no voters:
I20260619 07:02:00.428784 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4a7040df3c4edbb9facb771a330606" candidate_uuid: "e9ebefc474324be5935325d92a2a7dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bd90e650de470e9fdf3932416086cb" is_pre_election: true
I20260619 07:02:00.429039 29683 raft_consensus.cc:2454] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9ebefc474324be5935325d92a2a7dac in term 0.
I20260619 07:02:00.429082 29753 raft_consensus.cc:2781] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:02:00.429149 29753 raft_consensus.cc:482] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:02:00.429212 29753 raft_consensus.cc:3037] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:00.430346 29753 raft_consensus.cc:504] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } }
I20260619 07:02:00.430706 29753 leader_election.cc:288] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [CANDIDATE]: Term 1 election: Requested vote from peers 4af77b80e21c4f85a8c12991eb8e154d (127.11.106.65:35343), 21bd90e650de470e9fdf3932416086cb (127.11.106.67:38937)
I20260619 07:02:00.431016 29421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4a7040df3c4edbb9facb771a330606" candidate_uuid: "e9ebefc474324be5935325d92a2a7dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af77b80e21c4f85a8c12991eb8e154d"
I20260619 07:02:00.431051 29683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4a7040df3c4edbb9facb771a330606" candidate_uuid: "e9ebefc474324be5935325d92a2a7dac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bd90e650de470e9fdf3932416086cb"
I20260619 07:02:00.431129 29421 raft_consensus.cc:3037] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:00.431210 29683 raft_consensus.cc:3037] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:00.432367 29421 raft_consensus.cc:2454] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9ebefc474324be5935325d92a2a7dac in term 1.
I20260619 07:02:00.432456 29683 raft_consensus.cc:2454] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9ebefc474324be5935325d92a2a7dac in term 1.
I20260619 07:02:00.432729 29488 leader_election.cc:302] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af77b80e21c4f85a8c12991eb8e154d, e9ebefc474324be5935325d92a2a7dac; no voters:
I20260619 07:02:00.432940 29753 raft_consensus.cc:2781] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:00.433228 29753 raft_consensus.cc:686] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 1 LEADER]: Becoming Leader. State: Replica: e9ebefc474324be5935325d92a2a7dac, State: Running, Role: LEADER
I20260619 07:02:00.433585 29753 consensus_queue.cc:237] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } }
I20260619 07:02:00.436981 29278 catalog_manager.cc:5495] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac reported cstate change: term changed from 0 to 1, leader changed from <none> to e9ebefc474324be5935325d92a2a7dac (127.11.106.66). New cstate: current_term: 1 leader_uuid: "e9ebefc474324be5935325d92a2a7dac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9ebefc474324be5935325d92a2a7dac" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37357 } health_report { overall_health: HEALTHY } } }
I20260619 07:02:00.473188 29745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.469969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.11.106.66:37357)
0619 07:02:00.470114 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:00.470118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:00.470130 (+ 12us) server_negotiation.cc:407] Connection header received
0619 07:02:00.470204 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:00.470206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:00.470260 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:00.470349 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:00.470566 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.471062 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.471853 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:00.472052 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:00.472123 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:00.472368 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:00.472806 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:00.472911 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:00.472954 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 07:02:00.472998 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 07:02:00.478927 29421 raft_consensus.cc:1261] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d [term 1 FOLLOWER]: Refusing update from remote peer e9ebefc474324be5935325d92a2a7dac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:00.478948 29683 raft_consensus.cc:1261] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [term 1 FOLLOWER]: Refusing update from remote peer e9ebefc474324be5935325d92a2a7dac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:00.479651 29756 consensus_queue.cc:1035] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af77b80e21c4f85a8c12991eb8e154d" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:00.479876 29753 consensus_queue.cc:1035] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [LEADER]: Connected to new peer: Peer: permanent_uuid: "21bd90e650de470e9fdf3932416086cb" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:00.491652 29759 mvcc.cc:205] Tried to move back new op lower bound from 7298467923874570240 to 7298467923698831360. Current Snapshot: MvccSnapshot[applied={T|T < 7298467923874570240 or (T in {7298467923874570240})}]
I20260619 07:02:00.496419 29279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:00.496613 29279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:00.499107 29279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db6bd5d164ed40da88967a810c6c3222: Sending DeleteTablet for 3 replicas of tablet 2a4a7040df3c4edbb9facb771a330606
I20260619 07:02:00.499804 29401 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4a7040df3c4edbb9facb771a330606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:00 UTC) from {username='slave'} at 127.0.0.1:56674
I20260619 07:02:00.499949 29663 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4a7040df3c4edbb9facb771a330606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:00 UTC) from {username='slave'} at 127.0.0.1:38630
I20260619 07:02:00.500303 29769 tablet_replica.cc:330] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d: stopping tablet replica
I20260619 07:02:00.500420 29532 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4a7040df3c4edbb9facb771a330606 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:00 UTC) from {username='slave'} at 127.0.0.1:58632
I20260619 07:02:00.500801 29769 raft_consensus.cc:2229] T 2a4a7040df3c4edbb9facb771a330606 P 4af77b80e21c4f85a8c12991eb8e154d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:02:00.500898 29770 tablet_replica.cc:330] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: stopping tablet replica
I20260619 07:02:00.500980 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 29339
I20260619 07:02:00.501839 29771 tablet_replica.cc:330] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: stopping tablet replica
I20260619 07:02:00.502075 29771 raft_consensus.cc:2229] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 1 LEADER]: Raft consensus shutting down.
I20260619 07:02:00.502398 29771 raft_consensus.cc:2258] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:00.502723 29770 raft_consensus.cc:2229] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:02:00.503019 29770 raft_consensus.cc:2258] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:00.504657 29770 ts_tablet_manager.cc:1905] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:00.504657 29771 ts_tablet_manager.cc:1905] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:00.507344 29770 ts_tablet_manager.cc:1918] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:00.507342 29771 ts_tablet_manager.cc:1918] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:00.507439 29771 log.cc:1196] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/2a4a7040df3c4edbb9facb771a330606
I20260619 07:02:00.507550 29770 log.cc:1196] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/2a4a7040df3c4edbb9facb771a330606
I20260619 07:02:00.507771 29771 ts_tablet_manager.cc:1939] T 2a4a7040df3c4edbb9facb771a330606 P e9ebefc474324be5935325d92a2a7dac: Deleting consensus metadata
I20260619 07:02:00.507977 29770 ts_tablet_manager.cc:1939] T 2a4a7040df3c4edbb9facb771a330606 P 21bd90e650de470e9fdf3932416086cb: Deleting consensus metadata
I20260619 07:02:00.509099 29266 catalog_manager.cc:4841] TS 21bd90e650de470e9fdf3932416086cb (127.11.106.67:38937): tablet 2a4a7040df3c4edbb9facb771a330606 (table test-table [id=fd6089a058914a409223738c68fc93da]) successfully deleted
I20260619 07:02:00.509316 29266 catalog_manager.cc:4841] TS e9ebefc474324be5935325d92a2a7dac (127.11.106.66:37357): tablet 2a4a7040df3c4edbb9facb771a330606 (table test-table [id=fd6089a058914a409223738c68fc93da]) successfully deleted
W20260619 07:02:00.509836 29266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:35343 (error 108)
I20260619 07:02:00.510664 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 29470
I20260619 07:02:00.510875 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:00.510577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:35343 (local address 127.0.0.1:56686)
0619 07:02:00.510731 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:00.510798 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 07:02:00.511502 29266 catalog_manager.cc:4584] TS 4af77b80e21c4f85a8c12991eb8e154d (127.11.106.65:35343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a4a7040df3c4edbb9facb771a330606: Network error: Client connection negotiation failed: client connection to 127.11.106.65:35343: connect: Connection refused (error 111)
I20260619 07:02:00.516271 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 29601
I20260619 07:02:00.521711 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 29248
2026-06-19T07:02:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1102 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:00 dist-test-slave-10d0 krb5kdc[29775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:00 dist-test-slave-10d0 krb5kdc[29775](info): set up 2 sockets
Jun 19 07:02:00 dist-test-slave-10d0 krb5kdc[29775](info): commencing operation
krb5kdc: starting...
W20260619 07:02:01.565016 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:01 dist-test-slave-10d0 krb5kdc[29775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:01.684278 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45809
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:01.794759 29788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:01.795027 29788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:01.795075 29788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:01.799100 29788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:01.799153 29788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:01.799173 29788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:01.799190 29788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:01.799249 29788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:01.804426 29788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:45809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:01.805506 29788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:01.806517 29788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 07:02:01.814963 29788 server_base.cc:1034] running on GCE node
W20260619 07:02:01.814913 29794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:01.814950 29793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:01.814889 29796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:01.815585 29788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:01.816718 29788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:01.817903 29788 hybrid_clock.cc:648] HybridClock initialized: now 1781852521817883 us; error 31 us; skew 500 ppm
Jun 19 07:02:01 dist-test-slave-10d0 krb5kdc[29775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852521, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:01.820457 29788 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:01.821314 29788 webserver.cc:460] Webserver started at http://127.11.106.126:36023/ using document root <none> and password file <none>
I20260619 07:02:01.821861 29788 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:01.821952 29788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:01.822182 29788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:01.824167 29788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "c2262d28e22441efa10e6408e178661c"
format_stamp: "Formatted at 2026-06-19 07:02:01 on dist-test-slave-10d0"
server_key: "756ddc42a6261070bb403f0ac1d82756"
server_key_iv: "b6ece7dfe3e3a92b42b5cf67b864935c"
server_key_version: "encryptionkey@0"
I20260619 07:02:01.824781 29788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "c2262d28e22441efa10e6408e178661c"
format_stamp: "Formatted at 2026-06-19 07:02:01 on dist-test-slave-10d0"
server_key: "756ddc42a6261070bb403f0ac1d82756"
server_key_iv: "b6ece7dfe3e3a92b42b5cf67b864935c"
server_key_version: "encryptionkey@0"
I20260619 07:02:01.828382 29788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 07:02:01.830778 29803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:01.832089 29788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:01.832226 29788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "c2262d28e22441efa10e6408e178661c"
format_stamp: "Formatted at 2026-06-19 07:02:01 on dist-test-slave-10d0"
server_key: "756ddc42a6261070bb403f0ac1d82756"
server_key_iv: "b6ece7dfe3e3a92b42b5cf67b864935c"
server_key_version: "encryptionkey@0"
I20260619 07:02:01.832343 29788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:01.844918 29788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:01.847651 29788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:01.847849 29788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:01.854984 29788 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:45809
I20260619 07:02:01.855010 29855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:45809 every 8 connection(s)
I20260619 07:02:01.855924 29788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:01.858678 29856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:01.860369 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 29788
I20260619 07:02:01.860510 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:01.860831 11689 external_mini_cluster.cc:1428] Setting key 5f47f6688c0c3a5a916a1520ebf20d7c
I20260619 07:02:01.864933 29856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c: Bootstrap starting.
I20260619 07:02:01.867122 29856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:01.867851 29856 log.cc:824] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c: Log is configured to *not* fsync() on all Append() calls
Jun 19 07:02:01 dist-test-slave-10d0 krb5kdc[29775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:01.869789 29856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c: No bootstrap required, opened a new log
I20260619 07:02:01.874625 29856 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2262d28e22441efa10e6408e178661c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } }
I20260619 07:02:01.874898 29856 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:01.874955 29856 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2262d28e22441efa10e6408e178661c, State: Initialized, Role: FOLLOWER
I20260619 07:02:01.875483 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:01.862314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.11.106.126:45809)
0619 07:02:01.862734 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:01.862749 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:01.862784 (+ 35us) server_negotiation.cc:407] Connection header received
0619 07:02:01.863393 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:01.863409 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:01.863743 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:01.864089 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:01.865007 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:01.866003 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:01.866871 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:01.867405 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:01.867581 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:01.870216 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:01.870223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:01.870241 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:01.870272 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:01.872320 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:01.872830 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:01.872836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:01.872842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:01.872982 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:01.873249 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:01.873252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:01.873254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:01.873770 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:01.873982 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:01.874202 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 07:02:01.874496 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260619 07:02:01.875427 29856 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2262d28e22441efa10e6408e178661c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } }
I20260619 07:02:01.875581 29856 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:01.875649 29856 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:01.875749 29856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:01.876874 29856 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2262d28e22441efa10e6408e178661c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } }
I20260619 07:02:01.877219 29856 leader_election.cc:302] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2262d28e22441efa10e6408e178661c; no voters:
I20260619 07:02:01.877526 29856 leader_election.cc:288] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:01.877678 29861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:01.877928 29861 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [term 1 LEADER]: Becoming Leader. State: Replica: c2262d28e22441efa10e6408e178661c, State: Running, Role: LEADER
I20260619 07:02:01.878314 29861 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2262d28e22441efa10e6408e178661c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } }
I20260619 07:02:01.878800 29856 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:01.880139 29863 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2262d28e22441efa10e6408e178661c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2262d28e22441efa10e6408e178661c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } } }
I20260619 07:02:01.880295 29863 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:01.880120 29862 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2262d28e22441efa10e6408e178661c. Latest consensus state: current_term: 1 leader_uuid: "c2262d28e22441efa10e6408e178661c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2262d28e22441efa10e6408e178661c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 45809 } } }
I20260619 07:02:01.880530 29862 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:01.880798 29871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:01.883417 29871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:01.888465 29871 catalog_manager.cc:1347] Generated new cluster ID: d1897d84c4014c4baff47d17e306cbd7
I20260619 07:02:01.888535 29871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:01.914265 29871 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:01.915589 29871 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:01.925361 29871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c: Generated new TSK 0
I20260619 07:02:01.926095 29871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 07:02:01.963358 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:02.094554 29882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:02.094813 29882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:02.094856 29882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:02.098970 29882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:02.099025 29882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:02.099210 29882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:02:02.104327 29882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:02.105360 29882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:02.106361 29882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:02.115567 29890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:02.115599 29887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:02.115655 29882 server_base.cc:1034] running on GCE node
W20260619 07:02:02.115610 29888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:02.116263 29882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:02.116923 29882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:02.118083 29882 hybrid_clock.cc:648] HybridClock initialized: now 1781852522118067 us; error 27 us; skew 500 ppm
Jun 19 07:02:02 dist-test-slave-10d0 krb5kdc[29775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852522, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:02.124066 29882 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:02:02.125041 29882 webserver.cc:460] Webserver started at http://127.11.106.65:44781/ using document root <none> and password file <none>
I20260619 07:02:02.125612 29882 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:02.125672 29882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:02.125908 29882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:02.127938 29882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "5037d82dc3da40c4af94d606c97a1267"
format_stamp: "Formatted at 2026-06-19 07:02:02 on dist-test-slave-10d0"
server_key: "94aa9cc24cb285d502215b53bb781485"
server_key_iv: "1a4296b1510fb8dd8f67838bde19d1ad"
server_key_version: "encryptionkey@0"
I20260619 07:02:02.128479 29882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "5037d82dc3da40c4af94d606c97a1267"
format_stamp: "Formatted at 2026-06-19 07:02:02 on dist-test-slave-10d0"
server_key: "94aa9cc24cb285d502215b53bb781485"
server_key_iv: "1a4296b1510fb8dd8f67838bde19d1ad"
server_key_version: "encryptionkey@0"
I20260619 07:02:02.132138 29882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:02:02.134495 29897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:02.135691 29882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:02.135811 29882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "5037d82dc3da40c4af94d606c97a1267"
format_stamp: "Formatted at 2026-06-19 07:02:02 on dist-test-slave-10d0"
server_key: "94aa9cc24cb285d502215b53bb781485"
server_key_iv: "1a4296b1510fb8dd8f67838bde19d1ad"
server_key_version: "encryptionkey@0"
I20260619 07:02:02.135907 29882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:02.151922 29882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:02.154826 29882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:02.155027 29882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:02.155627 29882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:02.156445 29882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:02.156491 29882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:02.156579 29882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:02.156630 29882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:02.165340 29882 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:40909
I20260619 07:02:02.165357 30010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:40909 every 8 connection(s)
I20260619 07:02:02.166236 29882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:02:02.170495 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 29882
I20260619 07:02:02.170646 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 07:02:02.170955 11689 external_mini_cluster.cc:1428] Setting key be80b6e86698afff280b717991523eaf
Jun 19 07:02:02 dist-test-slave-10d0 krb5kdc[29775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852522, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 07:02:02.181480 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.168056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:41791 (local address 127.11.106.126:45809)
0619 07:02:02.168199 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.168203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.169151 (+ 948us) server_negotiation.cc:407] Connection header received
0619 07:02:02.170520 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.170523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.170583 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.170672 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:02.172108 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.172833 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.174556 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.174802 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.174918 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.177443 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:02.177452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:02.177454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:02.177476 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:02.179013 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:02.179682 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:02.179687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:02.179688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:02.179790 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:02.180220 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:02.180224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:02.180226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:02.180474 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:02.180617 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.181242 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.181355 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 07:02:02.182351 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.168457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45809 (local address 127.11.106.65:41791)
0619 07:02:02.168995 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:02.169039 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:02.170304 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:02.170880 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:02.170889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:02.171370 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:02.171949 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.171964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.172971 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.172977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.174430 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.174440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.174961 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.174965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.175139 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.175765 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:02.175782 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:02.177276 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:02.179169 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:02.179174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:02.179191 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:02.179574 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:02.179894 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:02.179897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:02.179899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:02.180115 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:02.180669 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:02.180678 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:02.181010 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 07:02:02.181179 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 07:02:02.181444 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 07:02:02.183750 30011 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45809
I20260619 07:02:02.184019 30011 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:02.184592 30011 heartbeater.cc:511] Master 127.11.106.126:45809 requested a full tablet report, sending...
I20260619 07:02:02.186255 29820 ts_manager.cc:194] Registered new tserver with Master: 5037d82dc3da40c4af94d606c97a1267 (127.11.106.65:40909)
I20260619 07:02:02.187965 29820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:41791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 07:02:02.200248 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:02.313629 30016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:02.313999 30016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:02.314086 30016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:02.318429 30016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:02.318492 30016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:02.318559 30016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:02:02.323815 30016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:02.324885 30016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:02.326026 30016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:02.335386 30022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:02.335377 30021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:02.335493 30016 server_base.cc:1034] running on GCE node
W20260619 07:02:02.335377 30024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:02.336059 30016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:02.336851 30016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:02.338011 30016 hybrid_clock.cc:648] HybridClock initialized: now 1781852522337982 us; error 41 us; skew 500 ppm
Jun 19 07:02:02 dist-test-slave-10d0 krb5kdc[29775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852522, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:02.340958 30016 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:02:02.341805 30016 webserver.cc:460] Webserver started at http://127.11.106.66:37877/ using document root <none> and password file <none>
I20260619 07:02:02.342360 30016 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:02.342418 30016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:02.342679 30016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:02.344715 30016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "179ff33af2a64a079743c28187fa4d73"
format_stamp: "Formatted at 2026-06-19 07:02:02 on dist-test-slave-10d0"
server_key: "45b44e1eeb4afffbfbdaf2350fffcd0a"
server_key_iv: "0091de687fac140bbd4fb88e17b02bcc"
server_key_version: "encryptionkey@0"
I20260619 07:02:02.345257 30016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "179ff33af2a64a079743c28187fa4d73"
format_stamp: "Formatted at 2026-06-19 07:02:02 on dist-test-slave-10d0"
server_key: "45b44e1eeb4afffbfbdaf2350fffcd0a"
server_key_iv: "0091de687fac140bbd4fb88e17b02bcc"
server_key_version: "encryptionkey@0"
I20260619 07:02:02.348929 30016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 07:02:02.351238 30031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:02.352515 30016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:02.352690 30016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "179ff33af2a64a079743c28187fa4d73"
format_stamp: "Formatted at 2026-06-19 07:02:02 on dist-test-slave-10d0"
server_key: "45b44e1eeb4afffbfbdaf2350fffcd0a"
server_key_iv: "0091de687fac140bbd4fb88e17b02bcc"
server_key_version: "encryptionkey@0"
I20260619 07:02:02.352814 30016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:02.370711 30016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:02.373579 30016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:02.373782 30016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:02.374354 30016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:02.375303 30016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:02.375348 30016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:02.375387 30016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:02.375411 30016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:02.384263 30016 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:44405
I20260619 07:02:02.384328 30144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:44405 every 8 connection(s)
I20260619 07:02:02.385215 30016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:02:02.386869 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 30016
I20260619 07:02:02.386992 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 07:02:02.387270 11689 external_mini_cluster.cc:1428] Setting key 6f9e6434c160d5d1d1f0d81f25d5e720
Jun 19 07:02:02 dist-test-slave-10d0 krb5kdc[29775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852522, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
I20260619 07:02:02.400099 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.387000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:58363 (local address 127.11.106.126:45809)
0619 07:02:02.387135 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.387139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.388046 (+ 907us) server_negotiation.cc:407] Connection header received
0619 07:02:02.389453 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.389456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.389510 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.389594 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:02.391150 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.391663 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.393506 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.393724 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.393803 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.396582 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:02.396596 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:02.396599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:02.396621 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:02.398015 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:02.398522 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:02.398525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:02.398526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:02.398603 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:02.398901 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:02.398903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:02.398904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:02.399140 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:02.399220 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.399826 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.399940 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 07:02:02.401062 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.387442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45809 (local address 127.11.106.66:58363)
0619 07:02:02.387893 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:02.387945 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:02.389244 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:02.389782 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:02.389789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:02.390274 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:02.390976 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.390997 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.391797 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.391804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.393376 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.393391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.393888 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.393896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.394135 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.394768 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:02.394789 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:02.396395 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:02.398156 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:02.398163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:02.398183 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:02.398429 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:02.398691 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:02.398693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:02.398694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:02.398823 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:02.399238 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:02.399243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:02.399472 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 07:02:02.399762 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 07:02:02.400124 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":125,"threads_started":1}
I20260619 07:02:02.402279 30145 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45809
I20260619 07:02:02.402531 30145 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:02.403026 30145 heartbeater.cc:511] Master 127.11.106.126:45809 requested a full tablet report, sending...
I20260619 07:02:02.404058 29820 ts_manager.cc:194] Registered new tserver with Master: 179ff33af2a64a079743c28187fa4d73 (127.11.106.66:44405)
I20260619 07:02:02.404711 29820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:58363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 07:02:02.415968 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:35721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:02.523545 30150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:02.523829 30150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:02.523916 30150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:02.527956 30150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:02.528035 30150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:02.528120 30150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:02:02.533308 30150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:35721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:45809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:02.534296 30150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:02.535369 30150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:02.545019 30158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:02.545073 30155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:02.545019 30156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:02.545255 30150 server_base.cc:1034] running on GCE node
I20260619 07:02:02.545701 30150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:02.546315 30150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:02.547503 30150 hybrid_clock.cc:648] HybridClock initialized: now 1781852522547494 us; error 28 us; skew 500 ppm
Jun 19 07:02:02 dist-test-slave-10d0 krb5kdc[29775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852522, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:02.550072 30150 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:02:02.550911 30150 webserver.cc:460] Webserver started at http://127.11.106.67:34531/ using document root <none> and password file <none>
I20260619 07:02:02.551458 30150 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:02.551548 30150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:02.551787 30150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:02.553783 30150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "059595edbab64bd99331f74fd27a5ad5"
format_stamp: "Formatted at 2026-06-19 07:02:02 on dist-test-slave-10d0"
server_key: "5ca2279d73ac3066508705a5033ee630"
server_key_iv: "ed8db1bd3598bf43e248726abbdfe5a1"
server_key_version: "encryptionkey@0"
I20260619 07:02:02.554332 30150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "059595edbab64bd99331f74fd27a5ad5"
format_stamp: "Formatted at 2026-06-19 07:02:02 on dist-test-slave-10d0"
server_key: "5ca2279d73ac3066508705a5033ee630"
server_key_iv: "ed8db1bd3598bf43e248726abbdfe5a1"
server_key_version: "encryptionkey@0"
I20260619 07:02:02.558019 30150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 07:02:02.560276 30165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:02.561456 30150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:02.561599 30150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "059595edbab64bd99331f74fd27a5ad5"
format_stamp: "Formatted at 2026-06-19 07:02:02 on dist-test-slave-10d0"
server_key: "5ca2279d73ac3066508705a5033ee630"
server_key_iv: "ed8db1bd3598bf43e248726abbdfe5a1"
server_key_version: "encryptionkey@0"
I20260619 07:02:02.561707 30150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:02.581131 30150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:02.583868 30150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:02.584064 30150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:02.584708 30150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:02.585530 30150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:02.585611 30150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:02.585680 30150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:02.585731 30150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:02.594271 30150 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:46219
I20260619 07:02:02.594309 30278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:46219 every 8 connection(s)
I20260619 07:02:02.595124 30150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:02:02.602276 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 30150
I20260619 07:02:02.602418 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 07:02:02.602725 11689 external_mini_cluster.cc:1428] Setting key 76880db759861a4c7aad2f8f2914cc1a
Jun 19 07:02:02 dist-test-slave-10d0 krb5kdc[29775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852522, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:02.608256 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.596840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:56479 (local address 127.11.106.126:45809)
0619 07:02:02.597000 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.597005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.597786 (+ 781us) server_negotiation.cc:407] Connection header received
0619 07:02:02.599034 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.599037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.599082 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.599161 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:02.600287 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.600811 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.601876 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.602197 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.602311 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.605076 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:02.605080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:02.605083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:02.605116 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:02.606319 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:02.606805 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:02.606809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:02.606811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:02.606894 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:02.607227 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:02.607231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:02.607232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:02.607405 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:02.607510 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.607966 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.608081 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 07:02:02.609072 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.597227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:45809 (local address 127.11.106.67:56479)
0619 07:02:02.597639 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:02.597680 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:02.598839 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:02.599322 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:02.599329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:02.599658 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:02.600118 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.600131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.600917 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.600921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.601772 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.601781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.602367 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.602372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.602569 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.603185 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:02.603210 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:02.604911 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:02.606434 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:02.606439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:02.606452 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:02.606702 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:02.606990 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:02.606993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:02.606995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:02.607128 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:02.607507 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:02.607513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:02.607746 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 07:02:02.607926 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 07:02:02.608223 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260619 07:02:02.610134 30279 heartbeater.cc:347] Connected to a master server at 127.11.106.126:45809
I20260619 07:02:02.610321 30279 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:02.610718 30279 heartbeater.cc:511] Master 127.11.106.126:45809 requested a full tablet report, sending...
I20260619 07:02:02.611629 29820 ts_manager.cc:194] Registered new tserver with Master: 059595edbab64bd99331f74fd27a5ad5 (127.11.106.67:46219)
I20260619 07:02:02.612239 29820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:56479
I20260619 07:02:02.616503 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:02:02.625875 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.618392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.11.106.126:45809)
0619 07:02:02.618537 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.618541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.618721 (+ 180us) server_negotiation.cc:407] Connection header received
0619 07:02:02.618865 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.618869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.618915 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.619000 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:02.619685 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.620181 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.621078 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.621292 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.621400 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.622887 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:02.622891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:02.622894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:02.622926 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:02.624066 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:02.624587 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:02.624592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:02.624594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:02.624670 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:02.625000 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:02.625004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:02.625005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:02.625206 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:02.625363 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.625628 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.625718 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 07:02:02.628643 29820 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:02:02.631507 29820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:02:02.646343 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.641047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49502 (local address 127.11.106.65:40909)
0619 07:02:02.641335 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.641352 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.641635 (+ 283us) server_negotiation.cc:407] Connection header received
0619 07:02:02.641719 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.641724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.641907 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.642075 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:02.642282 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.643363 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.644917 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.645714 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.645830 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.645973 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.646098 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.646197 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":116,"threads_started":1}
I20260619 07:02:02.646497 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.640589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:44405 (local address 127.0.0.1:58252)
0619 07:02:02.640864 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:02.640878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:02.640967 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:02.642483 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:02.642486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:02.642503 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:02.642604 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.642612 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.643545 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.643547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.645060 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.645068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.646085 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.646089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.646176 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.646197 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 07:02:02.646271 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 07:02:02.646326 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":58,"thread_start_us":84,"threads_started":1}
I20260619 07:02:02.646349 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.640639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:40909 (local address 127.0.0.1:49502)
0619 07:02:02.641427 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:02.641440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:02.641533 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:02.642039 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:02.642042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:02.642061 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:02.642162 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.642169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.643509 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.643514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.644778 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.644789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.645833 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.645835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.645911 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.645941 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 07:02:02.646061 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 07:02:02.646176 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":77,"thread_start_us":90,"threads_started":1}
I20260619 07:02:02.646901 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.640769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.11.106.66:44405)
0619 07:02:02.641589 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.641601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.641624 (+ 23us) server_negotiation.cc:407] Connection header received
0619 07:02:02.642163 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.642169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.642351 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.642503 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:02.642713 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.643416 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.645179 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.645959 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.646068 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.646204 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.646634 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.646744 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":94,"threads_started":1}
I20260619 07:02:02.647593 30290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.639865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:46219 (local address 127.0.0.1:37406)
0619 07:02:02.640225 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:02.640294 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:02.640431 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:02.641794 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:02.641800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:02.641823 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:02.641966 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.641977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.643473 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.643478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.645056 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.645068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.647230 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.647233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.647311 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.647328 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 07:02:02.647379 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 07:02:02.647458 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":137,"thread_start_us":62,"threads_started":1}
I20260619 07:02:02.647645 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.640967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.11.106.67:46219)
0619 07:02:02.641314 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.641326 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.641349 (+ 23us) server_negotiation.cc:407] Connection header received
0619 07:02:02.641425 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.641432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.641655 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.641818 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:02.642104 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.643310 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.646263 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.647125 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.647237 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.647364 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.647432 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.647516 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":172,"threads_started":1}
I20260619 07:02:02.649724 30079 tablet_service.cc:1467] Processing CreateTablet for tablet 59f9c01f27c647968c615098a5cff552 (DEFAULT_TABLE table=test-table [id=b1a797aa698b4cf3af6baa83520b8db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:02.650709 30079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f9c01f27c647968c615098a5cff552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:02.650936 29945 tablet_service.cc:1467] Processing CreateTablet for tablet 59f9c01f27c647968c615098a5cff552 (DEFAULT_TABLE table=test-table [id=b1a797aa698b4cf3af6baa83520b8db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:02.651970 29945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f9c01f27c647968c615098a5cff552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:02.651635 30213 tablet_service.cc:1467] Processing CreateTablet for tablet 59f9c01f27c647968c615098a5cff552 (DEFAULT_TABLE table=test-table [id=b1a797aa698b4cf3af6baa83520b8db6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:02.652607 30213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f9c01f27c647968c615098a5cff552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:02.657555 30296 tablet_bootstrap.cc:492] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: Bootstrap starting.
I20260619 07:02:02.658963 30297 tablet_bootstrap.cc:492] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267: Bootstrap starting.
I20260619 07:02:02.659771 30296 tablet_bootstrap.cc:654] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:02.660840 30298 tablet_bootstrap.cc:492] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: Bootstrap starting.
I20260619 07:02:02.661221 30297 tablet_bootstrap.cc:654] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:02.661224 30296 log.cc:824] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:02.662510 30297 log.cc:824] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:02.662925 30298 tablet_bootstrap.cc:654] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:02.663780 30296 tablet_bootstrap.cc:492] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: No bootstrap required, opened a new log
I20260619 07:02:02.663975 30296 ts_tablet_manager.cc:1397] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 07:02:02.664047 30298 log.cc:824] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:02.664475 30297 tablet_bootstrap.cc:492] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267: No bootstrap required, opened a new log
I20260619 07:02:02.664688 30297 ts_tablet_manager.cc:1397] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 07:02:02.665822 30298 tablet_bootstrap.cc:492] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: No bootstrap required, opened a new log
I20260619 07:02:02.666014 30298 ts_tablet_manager.cc:1397] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 07:02:02.667932 30296 raft_consensus.cc:348] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } }
I20260619 07:02:02.668211 30296 raft_consensus.cc:374] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:02.668268 30296 raft_consensus.cc:729] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179ff33af2a64a079743c28187fa4d73, State: Initialized, Role: FOLLOWER
I20260619 07:02:02.668351 30297 raft_consensus.cc:348] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } }
I20260619 07:02:02.668612 30297 raft_consensus.cc:374] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:02.668669 30297 raft_consensus.cc:729] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5037d82dc3da40c4af94d606c97a1267, State: Initialized, Role: FOLLOWER
I20260619 07:02:02.668747 30296 consensus_queue.cc:260] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } }
I20260619 07:02:02.669064 30297 consensus_queue.cc:260] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } }
I20260619 07:02:02.669179 30298 raft_consensus.cc:348] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } }
I20260619 07:02:02.669423 30298 raft_consensus.cc:374] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:02.669483 30298 raft_consensus.cc:729] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059595edbab64bd99331f74fd27a5ad5, State: Initialized, Role: FOLLOWER
I20260619 07:02:02.669896 30298 consensus_queue.cc:260] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } }
I20260619 07:02:02.670006 30297 ts_tablet_manager.cc:1428] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:02:02.670259 30296 ts_tablet_manager.cc:1428] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 07:02:02.670351 30011 heartbeater.cc:503] Master 127.11.106.126:45809 was elected leader, sending a full tablet report...
I20260619 07:02:02.670552 30145 heartbeater.cc:503] Master 127.11.106.126:45809 was elected leader, sending a full tablet report...
I20260619 07:02:02.670826 30298 ts_tablet_manager.cc:1428] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260619 07:02:02.671388 30279 heartbeater.cc:503] Master 127.11.106.126:45809 was elected leader, sending a full tablet report...
I20260619 07:02:02.815493 30304 raft_consensus.cc:482] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:02:02.815730 30304 raft_consensus.cc:504] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } }
I20260619 07:02:02.817014 30304 leader_election.cc:288] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 179ff33af2a64a079743c28187fa4d73 (127.11.106.66:44405), 5037d82dc3da40c4af94d606c97a1267 (127.11.106.65:40909)
I20260619 07:02:02.821935 30293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.817471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:51053 (local address 127.11.106.66:44405)
0619 07:02:02.817636 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.817641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.817660 (+ 19us) server_negotiation.cc:407] Connection header received
0619 07:02:02.817717 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.817720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.817776 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.817870 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:02.818229 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.819053 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.820526 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.821205 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.821351 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.821408 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.821727 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.821798 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 07:02:02.821934 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.817703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:60859 (local address 127.11.106.65:40909)
0619 07:02:02.817854 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.817860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.817982 (+ 122us) server_negotiation.cc:407] Connection header received
0619 07:02:02.818203 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.818209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.818263 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.818345 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:02.818615 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.819276 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.820672 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.821267 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.821371 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.821485 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.821727 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.821798 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 07:02:02.821951 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.817247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:44405 (local address 127.11.106.67:51053)
0619 07:02:02.817388 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:02.817406 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:02.817507 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:02.817912 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:02.817918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:02.817943 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:02.818092 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.818103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.819160 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.819162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.820390 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.820401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.821351 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.821355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.821582 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.821613 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 07:02:02.821695 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 07:02:02.821782 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":80}
I20260619 07:02:02.821960 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.817394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:40909 (local address 127.11.106.67:60859)
0619 07:02:02.817895 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:02.817908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:02.818039 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:02.818379 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:02.818381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:02.818397 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:02.818491 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.818495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.819370 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.819373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.820501 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:02.820512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.821362 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.821365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:02.821524 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.821613 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 07:02:02.821695 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 07:02:02.821782 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":79,"thread_start_us":165,"threads_started":1}
I20260619 07:02:02.822752 30099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f9c01f27c647968c615098a5cff552" candidate_uuid: "059595edbab64bd99331f74fd27a5ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff33af2a64a079743c28187fa4d73" is_pre_election: true
I20260619 07:02:02.822937 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f9c01f27c647968c615098a5cff552" candidate_uuid: "059595edbab64bd99331f74fd27a5ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5037d82dc3da40c4af94d606c97a1267" is_pre_election: true
I20260619 07:02:02.823088 30099 raft_consensus.cc:2454] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059595edbab64bd99331f74fd27a5ad5 in term 0.
I20260619 07:02:02.823263 29965 raft_consensus.cc:2454] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059595edbab64bd99331f74fd27a5ad5 in term 0.
I20260619 07:02:02.823596 30168 leader_election.cc:302] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059595edbab64bd99331f74fd27a5ad5, 179ff33af2a64a079743c28187fa4d73; no voters:
I20260619 07:02:02.823876 30304 raft_consensus.cc:2781] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:02:02.823967 30304 raft_consensus.cc:482] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:02:02.823998 30304 raft_consensus.cc:3037] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:02.825089 30304 raft_consensus.cc:504] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } }
I20260619 07:02:02.825454 30304 leader_election.cc:288] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [CANDIDATE]: Term 1 election: Requested vote from peers 179ff33af2a64a079743c28187fa4d73 (127.11.106.66:44405), 5037d82dc3da40c4af94d606c97a1267 (127.11.106.65:40909)
I20260619 07:02:02.825824 29965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f9c01f27c647968c615098a5cff552" candidate_uuid: "059595edbab64bd99331f74fd27a5ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5037d82dc3da40c4af94d606c97a1267"
I20260619 07:02:02.825809 30099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f9c01f27c647968c615098a5cff552" candidate_uuid: "059595edbab64bd99331f74fd27a5ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "179ff33af2a64a079743c28187fa4d73"
I20260619 07:02:02.825929 29965 raft_consensus.cc:3037] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:02.825960 30099 raft_consensus.cc:3037] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:02.827180 29965 raft_consensus.cc:2454] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059595edbab64bd99331f74fd27a5ad5 in term 1.
I20260619 07:02:02.827217 30099 raft_consensus.cc:2454] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059595edbab64bd99331f74fd27a5ad5 in term 1.
I20260619 07:02:02.827483 30167 leader_election.cc:302] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059595edbab64bd99331f74fd27a5ad5, 5037d82dc3da40c4af94d606c97a1267; no voters:
I20260619 07:02:02.827672 30304 raft_consensus.cc:2781] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:02.827911 30304 raft_consensus.cc:686] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 1 LEADER]: Becoming Leader. State: Replica: 059595edbab64bd99331f74fd27a5ad5, State: Running, Role: LEADER
I20260619 07:02:02.828280 30304 consensus_queue.cc:237] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } }
I20260619 07:02:02.831578 29819 catalog_manager.cc:5495] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059595edbab64bd99331f74fd27a5ad5 (127.11.106.67). New cstate: current_term: 1 leader_uuid: "059595edbab64bd99331f74fd27a5ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059595edbab64bd99331f74fd27a5ad5" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 } health_report { overall_health: UNKNOWN } } }
W20260619 07:02:02.845955 30280 tablet.cc:2369] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:02:02.881527 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.878020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.11.106.67:46219)
0619 07:02:02.878166 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:02.878171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:02.878184 (+ 13us) server_negotiation.cc:407] Connection header received
0619 07:02:02.878254 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:02.878257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:02.878310 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:02.878400 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:02.878609 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.879085 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.880054 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:02.880273 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:02.880349 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:02.880750 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:02.881153 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:02.881262 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:02.881366 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 07:02:02.881401 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 07:02:02.886358 30146 tablet.cc:2369] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:02:02.887231 30099 raft_consensus.cc:1261] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [term 1 FOLLOWER]: Refusing update from remote peer 059595edbab64bd99331f74fd27a5ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:02.887233 29965 raft_consensus.cc:1261] T 59f9c01f27c647968c615098a5cff552 P 5037d82dc3da40c4af94d606c97a1267 [term 1 FOLLOWER]: Refusing update from remote peer 059595edbab64bd99331f74fd27a5ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:02.887938 30306 consensus_queue.cc:1035] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5037d82dc3da40c4af94d606c97a1267" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:02.888111 30304 consensus_queue.cc:1035] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "179ff33af2a64a079743c28187fa4d73" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:02.893942 30309 mvcc.cc:205] Tried to move back new op lower bound from 7298467933739020288 to 7298467933507481600. Current Snapshot: MvccSnapshot[applied={T|T < 7298467933739020288}]
I20260619 07:02:02.894431 30310 mvcc.cc:205] Tried to move back new op lower bound from 7298467933739020288 to 7298467933507481600. Current Snapshot: MvccSnapshot[applied={T|T < 7298467933739020288}]
I20260619 07:02:02.904878 29818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:02.905064 29818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:02.907303 29818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2262d28e22441efa10e6408e178661c: Sending DeleteTablet for 3 replicas of tablet 59f9c01f27c647968c615098a5cff552
I20260619 07:02:02.908113 29945 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f9c01f27c647968c615098a5cff552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:02 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:49502
I20260619 07:02:02.908161 30213 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f9c01f27c647968c615098a5cff552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:02 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:37406
I20260619 07:02:02.908170 30079 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f9c01f27c647968c615098a5cff552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:02 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:58252
I20260619 07:02:02.908609 30320 tablet_replica.cc:330] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: stopping tablet replica
I20260619 07:02:02.908905 30320 raft_consensus.cc:2229] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:02:02.908931 30318 tablet_replica.cc:330] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: stopping tablet replica
I20260619 07:02:02.909175 30318 raft_consensus.cc:2229] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 07:02:02.909219 30320 raft_consensus.cc:2258] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:02.909305 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 29882
I20260619 07:02:02.909628 30318 raft_consensus.cc:2258] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:02.910989 30318 ts_tablet_manager.cc:1905] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:02.910989 30320 ts_tablet_manager.cc:1905] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:02.913645 30318 ts_tablet_manager.cc:1918] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:02.913645 30320 ts_tablet_manager.cc:1918] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:02.913731 30318 log.cc:1196] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/59f9c01f27c647968c615098a5cff552
I20260619 07:02:02.913731 30320 log.cc:1196] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/59f9c01f27c647968c615098a5cff552
I20260619 07:02:02.914045 30318 ts_tablet_manager.cc:1939] T 59f9c01f27c647968c615098a5cff552 P 059595edbab64bd99331f74fd27a5ad5: Deleting consensus metadata
I20260619 07:02:02.914045 30320 ts_tablet_manager.cc:1939] T 59f9c01f27c647968c615098a5cff552 P 179ff33af2a64a079743c28187fa4d73: Deleting consensus metadata
I20260619 07:02:02.915153 29806 catalog_manager.cc:4841] TS 179ff33af2a64a079743c28187fa4d73 (127.11.106.66:44405): tablet 59f9c01f27c647968c615098a5cff552 (table test-table [id=b1a797aa698b4cf3af6baa83520b8db6]) successfully deleted
I20260619 07:02:02.915349 29806 catalog_manager.cc:4841] TS 059595edbab64bd99331f74fd27a5ad5 (127.11.106.67:46219): tablet 59f9c01f27c647968c615098a5cff552 (table test-table [id=b1a797aa698b4cf3af6baa83520b8db6]) successfully deleted
W20260619 07:02:02.919667 29805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:40909 (error 108)
I20260619 07:02:02.919767 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 30016
I20260619 07:02:02.922147 30290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:02.921782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:40909 (local address 127.0.0.1:49504)
0619 07:02:02.921951 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:02.922033 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:40909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 07:02:02.922322 29805 catalog_manager.cc:4584] TS 5037d82dc3da40c4af94d606c97a1267 (127.11.106.65:40909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59f9c01f27c647968c615098a5cff552: Network error: Client connection negotiation failed: client connection to 127.11.106.65:40909: connect: Connection refused (error 111)
I20260619 07:02:02.925709 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 30150
I20260619 07:02:02.930645 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 29788
2026-06-19T07:02:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2409 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T07:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:02Z Disabled control of system clock
I20260619 07:02:02.959755 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:43819
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:43819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 07:02:03.068672 30325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:03.068909 30325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:03.068979 30325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:03.072907 30325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:03.072958 30325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:03.072974 30325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:03.073017 30325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:03.073048 30325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:03.077845 30325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:43819
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:43819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:03.078908 30325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:03.080310 30325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:03.088737 30333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:03.088784 30330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:03.088754 30331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:03.088951 30325 server_base.cc:1034] running on GCE node
I20260619 07:02:03.089504 30325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:03.090601 30325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:03.091763 30325 hybrid_clock.cc:648] HybridClock initialized: now 1781852523091731 us; error 43 us; skew 500 ppm
I20260619 07:02:03.093626 30325 webserver.cc:460] Webserver started at http://127.11.106.126:45021/ using document root <none> and password file <none>
I20260619 07:02:03.094202 30325 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:03.094264 30325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:03.094497 30325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:03.096479 30325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "647509b01c7a4ec395ab3c22bb7765b2"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "9d2ba18658e89e76ae46838c7352d674"
server_key_iv: "47c6e680ba3595424c88212c2e8b3fd2"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.097091 30325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "647509b01c7a4ec395ab3c22bb7765b2"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "9d2ba18658e89e76ae46838c7352d674"
server_key_iv: "47c6e680ba3595424c88212c2e8b3fd2"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.100736 30325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 07:02:03.103083 30339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.104238 30325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:03.104373 30325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "647509b01c7a4ec395ab3c22bb7765b2"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "9d2ba18658e89e76ae46838c7352d674"
server_key_iv: "47c6e680ba3595424c88212c2e8b3fd2"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.104476 30325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:03.121928 30325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:03.122673 30325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:03.122879 30325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:03.130086 30325 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:43819
I20260619 07:02:03.130105 30391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:43819 every 8 connection(s)
I20260619 07:02:03.131021 30325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:03.134038 30392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:03.135419 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 30325
I20260619 07:02:03.135542 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:03.135877 11689 external_mini_cluster.cc:1428] Setting key b7018bac72c2b45c846ca9a65978fc5e
I20260619 07:02:03.140210 30392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2: Bootstrap starting.
I20260619 07:02:03.142861 30392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:03.143657 30392 log.cc:824] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:03.145079 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.137276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.11.106.126:43819)
0619 07:02:03.137862 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.137877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.137912 (+ 35us) server_negotiation.cc:407] Connection header received
0619 07:02:03.138564 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.138580 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.138956 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.139226 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:03.139540 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.140651 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.141842 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.142278 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.142388 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.143235 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:03.143240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:03.143252 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:03.143624 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:03.143680 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:03.143699 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:03.143809 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.143964 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.144320 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":121,"threads_started":1}
I20260619 07:02:03.145694 30392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2: No bootstrap required, opened a new log
I20260619 07:02:03.149521 30392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647509b01c7a4ec395ab3c22bb7765b2" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43819 } }
I20260619 07:02:03.149760 30392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:03.149837 30392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 647509b01c7a4ec395ab3c22bb7765b2, State: Initialized, Role: FOLLOWER
I20260619 07:02:03.150246 30392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647509b01c7a4ec395ab3c22bb7765b2" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43819 } }
I20260619 07:02:03.150393 30392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:03.150472 30392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:03.150580 30392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:03.151608 30392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647509b01c7a4ec395ab3c22bb7765b2" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43819 } }
I20260619 07:02:03.151939 30392 leader_election.cc:302] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 647509b01c7a4ec395ab3c22bb7765b2; no voters:
I20260619 07:02:03.152269 30392 leader_election.cc:288] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:03.152390 30397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:03.152674 30397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [term 1 LEADER]: Becoming Leader. State: Replica: 647509b01c7a4ec395ab3c22bb7765b2, State: Running, Role: LEADER
I20260619 07:02:03.152961 30397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647509b01c7a4ec395ab3c22bb7765b2" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43819 } }
I20260619 07:02:03.153477 30392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:03.154523 30398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "647509b01c7a4ec395ab3c22bb7765b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647509b01c7a4ec395ab3c22bb7765b2" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43819 } } }
I20260619 07:02:03.154556 30399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 647509b01c7a4ec395ab3c22bb7765b2. Latest consensus state: current_term: 1 leader_uuid: "647509b01c7a4ec395ab3c22bb7765b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "647509b01c7a4ec395ab3c22bb7765b2" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43819 } } }
I20260619 07:02:03.154680 30398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:03.154683 30399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:03.155104 30406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:03.157878 30406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:03.163473 30406 catalog_manager.cc:1347] Generated new cluster ID: 9f3ba421949940a6b5f57f73e144a44e
I20260619 07:02:03.163554 30406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:03.177996 30406 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:03.178938 30406 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:03.186877 30406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2: Generated new TSK 0
I20260619 07:02:03.187543 30406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:03.198305 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:43819
--builtin_ntp_servers=127.11.106.84:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 07:02:03.308228 30416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:03.308490 30416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:03.308578 30416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:03.312681 30416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:03.312737 30416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:03.312793 30416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:02:03.317629 30416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:43819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:03.318791 30416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:03.320236 30416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:03.330134 30421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:03.330197 30422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:03.330274 30416 server_base.cc:1034] running on GCE node
W20260619 07:02:03.330124 30424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:03.330747 30416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:03.331435 30416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:03.332623 30416 hybrid_clock.cc:648] HybridClock initialized: now 1781852523332606 us; error 31 us; skew 500 ppm
I20260619 07:02:03.334425 30416 webserver.cc:460] Webserver started at http://127.11.106.65:36825/ using document root <none> and password file <none>
I20260619 07:02:03.335012 30416 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:03.335105 30416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:03.335345 30416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:03.337527 30416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "7c8d387afa474394bfaf0344084dbea5"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "a92796fa189bb2e93330402958aa14b0"
server_key_iv: "42a8c376146c9fdacfe446577b405bc4"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.338099 30416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "7c8d387afa474394bfaf0344084dbea5"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "a92796fa189bb2e93330402958aa14b0"
server_key_iv: "42a8c376146c9fdacfe446577b405bc4"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.341876 30416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:02:03.344200 30430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.345420 30416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:03.345566 30416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "7c8d387afa474394bfaf0344084dbea5"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "a92796fa189bb2e93330402958aa14b0"
server_key_iv: "42a8c376146c9fdacfe446577b405bc4"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.345682 30416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:03.359859 30416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:03.360507 30416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:03.360733 30416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:03.361300 30416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:03.362216 30416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:03.362288 30416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.362361 30416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:03.362416 30416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.371064 30416 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:37163
I20260619 07:02:03.371099 30543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:37163 every 8 connection(s)
I20260619 07:02:03.371960 30416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:02:03.374843 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 30416
I20260619 07:02:03.374982 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 07:02:03.375308 11689 external_mini_cluster.cc:1428] Setting key 830dbcd032b198c3191a6a0372803e9a
I20260619 07:02:03.377305 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:43819
--builtin_ntp_servers=127.11.106.84:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 07:02:03.381681 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.373721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:55373 (local address 127.11.106.126:43819)
0619 07:02:03.373859 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.373863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.375008 (+ 1145us) server_negotiation.cc:407] Connection header received
0619 07:02:03.376639 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.376643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.376704 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.376762 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:03.377912 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.378460 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.379610 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.379844 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.379957 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.381165 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:03.381169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:03.381171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:03.381185 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:03.381212 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:03.381220 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:03.381298 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.381461 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.381530 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 07:02:03.382639 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.374050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:43819 (local address 127.11.106.65:55373)
0619 07:02:03.374821 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:03.374879 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:03.376392 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:03.377116 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:03.377122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:03.377202 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:03.377764 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.377778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.378592 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.378596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.379471 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.379480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.379995 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.379999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.380169 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.380858 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 07:02:03.380859 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:03.380899 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 07:02:03.380983 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:03.381300 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:03.381312 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:03.381320 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 07:02:03.381459 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 07:02:03.381734 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":106,"threads_started":1}
I20260619 07:02:03.383831 30544 heartbeater.cc:347] Connected to a master server at 127.11.106.126:43819
I20260619 07:02:03.384096 30544 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:03.384809 30544 heartbeater.cc:511] Master 127.11.106.126:43819 requested a full tablet report, sending...
I20260619 07:02:03.386377 30356 ts_manager.cc:194] Registered new tserver with Master: 7c8d387afa474394bfaf0344084dbea5 (127.11.106.65:37163)
I20260619 07:02:03.387491 30356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.65:55373
W20260619 07:02:03.487545 30547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:03.487795 30547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:03.487874 30547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:03.491847 30547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:03.491899 30547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:03.492020 30547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:02:03.496640 30547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:43819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:03.497592 30547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:03.498893 30547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:03.507989 30552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:03.508023 30555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:03.508039 30553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:03.508327 30547 server_base.cc:1034] running on GCE node
I20260619 07:02:03.508919 30547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:03.509567 30547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:03.510746 30547 hybrid_clock.cc:648] HybridClock initialized: now 1781852523510717 us; error 51 us; skew 500 ppm
I20260619 07:02:03.512427 30547 webserver.cc:460] Webserver started at http://127.11.106.66:45363/ using document root <none> and password file <none>
I20260619 07:02:03.513033 30547 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:03.513132 30547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:03.513361 30547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:03.515327 30547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "97e9505461e5d00b346ebc3b89c61d0f"
server_key_iv: "8d4142bb3f7624e2b46cd9ed141dea93"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.515873 30547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "97e9505461e5d00b346ebc3b89c61d0f"
server_key_iv: "8d4142bb3f7624e2b46cd9ed141dea93"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.519400 30547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 07:02:03.521755 30561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.522872 30547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:03.523002 30547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "97e9505461e5d00b346ebc3b89c61d0f"
server_key_iv: "8d4142bb3f7624e2b46cd9ed141dea93"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.523119 30547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:03.537557 30547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:03.538182 30547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:03.538367 30547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:03.538908 30547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:03.539793 30547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:03.539865 30547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.539932 30547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:03.539983 30547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.548694 30547 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:45533
I20260619 07:02:03.548750 30674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:45533 every 8 connection(s)
I20260619 07:02:03.549713 30547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:02:03.553095 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 30547
I20260619 07:02:03.553226 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 07:02:03.553541 11689 external_mini_cluster.cc:1428] Setting key bdc37a7e4bcffa211e449611a3ec3725
I20260619 07:02:03.555860 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:43819
--builtin_ntp_servers=127.11.106.84:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 07:02:03.560002 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.551493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:58981 (local address 127.11.106.126:43819)
0619 07:02:03.551637 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.551641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.552711 (+ 1070us) server_negotiation.cc:407] Connection header received
0619 07:02:03.554531 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.554535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.554614 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.554697 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:03.555806 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.556402 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.557871 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.558256 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.558381 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.559414 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:03.559421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:03.559425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:03.559442 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:03.559467 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:03.559478 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:03.559558 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.559718 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.559794 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 07:02:03.560756 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.551876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:43819 (local address 127.11.106.66:58981)
0619 07:02:03.552514 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:03.552610 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:03.554309 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:03.554878 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:03.554887 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:03.554996 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:03.555644 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.555666 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.556531 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.556562 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.557747 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.557758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.558403 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.558406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.558542 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.559153 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 07:02:03.559154 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:03.559193 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 07:02:03.559257 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:03.559570 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:03.559578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:03.559583 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 07:02:03.559685 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 07:02:03.559896 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260619 07:02:03.561856 30675 heartbeater.cc:347] Connected to a master server at 127.11.106.126:43819
I20260619 07:02:03.562117 30675 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:03.562762 30675 heartbeater.cc:511] Master 127.11.106.126:43819 requested a full tablet report, sending...
I20260619 07:02:03.563889 30356 ts_manager.cc:194] Registered new tserver with Master: 68caa0156f8c4f65b2aabf07b8a9b1e4 (127.11.106.66:45533)
I20260619 07:02:03.564451 30356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.66:58981
W20260619 07:02:03.670589 30678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:03.670842 30678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:03.670924 30678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:03.675034 30678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:03.675096 30678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:03.675199 30678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:02:03.679922 30678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:43819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:03.681023 30678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:03.682603 30678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:03.692166 30683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:03.692209 30686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:03.692157 30684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:03.692400 30678 server_base.cc:1034] running on GCE node
I20260619 07:02:03.692904 30678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:03.693532 30678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:03.694743 30678 hybrid_clock.cc:648] HybridClock initialized: now 1781852523694701 us; error 48 us; skew 500 ppm
I20260619 07:02:03.696612 30678 webserver.cc:460] Webserver started at http://127.11.106.67:39765/ using document root <none> and password file <none>
I20260619 07:02:03.697213 30678 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:03.697314 30678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:03.697561 30678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:03.699604 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "68a9953aaf1a4ec39f9f516d553aef2b"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "e2c4b04a80660ffb194ae6141b648faf"
server_key_iv: "1ca0693fa0e9e6309df87c9d5de0f192"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.700260 30678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "68a9953aaf1a4ec39f9f516d553aef2b"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "e2c4b04a80660ffb194ae6141b648faf"
server_key_iv: "1ca0693fa0e9e6309df87c9d5de0f192"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.704037 30678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:02:03.706862 30692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.708082 30678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 07:02:03.708230 30678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "68a9953aaf1a4ec39f9f516d553aef2b"
format_stamp: "Formatted at 2026-06-19 07:02:03 on dist-test-slave-10d0"
server_key: "e2c4b04a80660ffb194ae6141b648faf"
server_key_iv: "1ca0693fa0e9e6309df87c9d5de0f192"
server_key_version: "encryptionkey@0"
I20260619 07:02:03.708369 30678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:03.722237 30678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:03.722887 30678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:03.723093 30678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:03.723671 30678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:03.724603 30678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:03.724647 30678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.724716 30678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:03.724759 30678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:03.733788 30678 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:38523
I20260619 07:02:03.733812 30805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:38523 every 8 connection(s)
I20260619 07:02:03.734680 30678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:02:03.742321 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 30678
I20260619 07:02:03.742460 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 07:02:03.742729 11689 external_mini_cluster.cc:1428] Setting key c8ee9a60aa4c25d13360cc3e314ea585
I20260619 07:02:03.744187 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.736628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:59853 (local address 127.11.106.126:43819)
0619 07:02:03.736797 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.736801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.737782 (+ 981us) server_negotiation.cc:407] Connection header received
0619 07:02:03.739168 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.739170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.739216 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.739288 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:03.740151 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.740691 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.742135 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.742354 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.742479 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.743595 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:03.743599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:03.743602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:03.743639 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:03.743665 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:03.743679 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:03.743753 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.743931 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.744010 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 07:02:03.745288 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.737000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:43819 (local address 127.11.106.67:59853)
0619 07:02:03.737659 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:03.737698 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:03.738975 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:03.739452 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:03.739458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:03.739542 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:03.740029 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.740042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.740798 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.740802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.741980 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.741990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.742519 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.742523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.742704 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.743333 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 07:02:03.743334 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:03.743373 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 07:02:03.743434 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:03.743772 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:03.743782 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:03.743789 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 07:02:03.743927 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 07:02:03.744226 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260619 07:02:03.746552 30806 heartbeater.cc:347] Connected to a master server at 127.11.106.126:43819
I20260619 07:02:03.746799 30806 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:03.747403 30806 heartbeater.cc:511] Master 127.11.106.126:43819 requested a full tablet report, sending...
I20260619 07:02:03.748497 30356 ts_manager.cc:194] Registered new tserver with Master: 68a9953aaf1a4ec39f9f516d553aef2b (127.11.106.67:38523)
I20260619 07:02:03.749208 30356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.67:59853
I20260619 07:02:03.756708 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:02:03.762241 30395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.758612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.11.106.126:43819)
0619 07:02:03.758727 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.758731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.758929 (+ 198us) server_negotiation.cc:407] Connection header received
0619 07:02:03.759025 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.759027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.759064 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.759132 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:03.759376 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.759878 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.760737 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.760967 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.761059 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.761832 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:03.761835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:03.761837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:03.761849 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:03.761868 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:03.761876 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:03.761986 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.762098 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.762132 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 07:02:03.765216 30356 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:02:03.768167 30356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:02:03.782723 30821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.777389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:38523 (local address 127.0.0.1:36796)
0619 07:02:03.777862 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:03.777876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:03.777958 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:03.778776 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:03.778779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:03.778799 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:03.778886 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.778892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.780155 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.780160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.781346 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.781355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.782150 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.782154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.782252 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.782288 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 07:02:03.782411 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 07:02:03.782531 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":86,"thread_start_us":91,"threads_started":1}
I20260619 07:02:03.782766 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.776620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:37163 (local address 127.0.0.1:36656)
0619 07:02:03.777020 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:03.777224 (+ 204us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:03.777357 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:03.778196 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:03.778202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:03.778231 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:03.778366 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.778372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.779916 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.779920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.781350 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.781362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.782433 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.782437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.782525 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.782545 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 07:02:03.782606 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 07:02:03.782659 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":96,"thread_start_us":111,"threads_started":1}
I20260619 07:02:03.783094 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.776767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.11.106.65:37163)
0619 07:02:03.777656 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.777670 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.777698 (+ 28us) server_negotiation.cc:407] Connection header received
0619 07:02:03.777782 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.777788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.778002 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.778507 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:03.778663 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.779799 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.781498 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.782281 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.782432 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.782611 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.782664 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.782774 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":118,"threads_started":1}
I20260619 07:02:03.783309 30819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.777109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:45533 (local address 127.0.0.1:49280)
0619 07:02:03.777591 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:03.777604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:03.777688 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:03.778607 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:03.778610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:03.778627 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:03.778712 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.778718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.779734 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.779739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.781356 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.781362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.782879 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.782883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.782990 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.783024 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 07:02:03.783092 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 07:02:03.783152 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":132,"thread_start_us":97,"threads_started":1}
I20260619 07:02:03.783418 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.777295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.11.106.66:45533)
0619 07:02:03.778165 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.778178 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.778201 (+ 23us) server_negotiation.cc:407] Connection header received
0619 07:02:03.778280 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.778286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.778463 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.778617 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:03.778821 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.779612 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.781498 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.782281 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.782432 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.782605 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.783156 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.783268 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":99,"threads_started":1}
I20260619 07:02:03.783766 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.777779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.11.106.67:38523)
0619 07:02:03.778211 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.778222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.778240 (+ 18us) server_negotiation.cc:407] Connection header received
0619 07:02:03.778301 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.778305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.778463 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.778617 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:03.779013 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.780051 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.781501 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.782028 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.782151 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.782348 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.783346 (+ 998us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.783612 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":133,"threads_started":1}
I20260619 07:02:03.786166 30740 tablet_service.cc:1467] Processing CreateTablet for tablet 18d70e873ed643a8b89cb6906f7b5e4d (DEFAULT_TABLE table=test-table [id=8e21a8af9dd6488daf55ca1bd7fe2999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:03.786259 30478 tablet_service.cc:1467] Processing CreateTablet for tablet 18d70e873ed643a8b89cb6906f7b5e4d (DEFAULT_TABLE table=test-table [id=8e21a8af9dd6488daf55ca1bd7fe2999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:03.786262 30609 tablet_service.cc:1467] Processing CreateTablet for tablet 18d70e873ed643a8b89cb6906f7b5e4d (DEFAULT_TABLE table=test-table [id=8e21a8af9dd6488daf55ca1bd7fe2999]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:03.787168 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d70e873ed643a8b89cb6906f7b5e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:03.787271 30478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d70e873ed643a8b89cb6906f7b5e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:03.787298 30609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18d70e873ed643a8b89cb6906f7b5e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:03.793638 30823 tablet_bootstrap.cc:492] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: Bootstrap starting.
I20260619 07:02:03.794870 30824 tablet_bootstrap.cc:492] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5: Bootstrap starting.
I20260619 07:02:03.796224 30823 tablet_bootstrap.cc:654] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:03.794395 30825 tablet_bootstrap.cc:492] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: Bootstrap starting.
I20260619 07:02:03.796942 30824 tablet_bootstrap.cc:654] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:03.797135 30823 log.cc:824] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:03.797742 30824 log.cc:824] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:03.798550 30825 tablet_bootstrap.cc:654] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:03.799243 30825 log.cc:824] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:03.799432 30823 tablet_bootstrap.cc:492] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: No bootstrap required, opened a new log
I20260619 07:02:03.799635 30823 ts_tablet_manager.cc:1397] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 07:02:03.800201 30824 tablet_bootstrap.cc:492] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5: No bootstrap required, opened a new log
I20260619 07:02:03.800386 30824 ts_tablet_manager.cc:1397] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 07:02:03.801047 30825 tablet_bootstrap.cc:492] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: No bootstrap required, opened a new log
I20260619 07:02:03.801237 30825 ts_tablet_manager.cc:1397] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 07:02:03.803987 30823 raft_consensus.cc:348] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } }
I20260619 07:02:03.804262 30823 raft_consensus.cc:374] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:03.804320 30823 raft_consensus.cc:729] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a9953aaf1a4ec39f9f516d553aef2b, State: Initialized, Role: FOLLOWER
I20260619 07:02:03.804208 30825 raft_consensus.cc:348] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } }
I20260619 07:02:03.804507 30825 raft_consensus.cc:374] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:03.804586 30825 raft_consensus.cc:729] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68caa0156f8c4f65b2aabf07b8a9b1e4, State: Initialized, Role: FOLLOWER
I20260619 07:02:03.804582 30824 raft_consensus.cc:348] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } }
I20260619 07:02:03.804831 30824 raft_consensus.cc:374] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:03.804888 30824 raft_consensus.cc:729] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8d387afa474394bfaf0344084dbea5, State: Initialized, Role: FOLLOWER
I20260619 07:02:03.804845 30823 consensus_queue.cc:260] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } }
I20260619 07:02:03.805045 30825 consensus_queue.cc:260] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } }
I20260619 07:02:03.805661 30806 heartbeater.cc:503] Master 127.11.106.126:43819 was elected leader, sending a full tablet report...
I20260619 07:02:03.805872 30823 ts_tablet_manager.cc:1428] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 07:02:03.805977 30825 ts_tablet_manager.cc:1428] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:02:03.806298 30675 heartbeater.cc:503] Master 127.11.106.126:43819 was elected leader, sending a full tablet report...
I20260619 07:02:03.806780 30824 consensus_queue.cc:260] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } }
I20260619 07:02:03.807379 30544 heartbeater.cc:503] Master 127.11.106.126:43819 was elected leader, sending a full tablet report...
I20260619 07:02:03.807715 30824 ts_tablet_manager.cc:1428] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260619 07:02:03.868734 30831 raft_consensus.cc:482] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:02:03.868971 30831 raft_consensus.cc:504] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } }
I20260619 07:02:03.870191 30831 leader_election.cc:288] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68a9953aaf1a4ec39f9f516d553aef2b (127.11.106.67:38523), 68caa0156f8c4f65b2aabf07b8a9b1e4 (127.11.106.66:45533)
W20260619 07:02:03.873276 30545 tablet.cc:2369] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:02:03.874534 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.870506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:38523 (local address 127.11.106.65:46977)
0619 07:02:03.870677 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:03.870694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:03.870812 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:03.871147 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:03.871150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:03.871188 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:03.871341 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.871348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.872290 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.872296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.873281 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.873291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.874123 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.874125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.874199 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.874219 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 07:02:03.874305 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 07:02:03.874376 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":57,"spinlock_wait_cycles":5120}
I20260619 07:02:03.874544 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.870716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:46977 (local address 127.11.106.67:38523)
0619 07:02:03.870862 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.870869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.870889 (+ 20us) server_negotiation.cc:407] Connection header received
0619 07:02:03.870954 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.870957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.871007 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.871097 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:03.871466 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.872173 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.873414 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.874006 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.874127 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.874179 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.874333 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.874409 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 07:02:03.874775 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.870617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:48845 (local address 127.11.106.66:45533)
0619 07:02:03.870751 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.870755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.871179 (+ 424us) server_negotiation.cc:407] Connection header received
0619 07:02:03.871277 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.871279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.871342 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.871420 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:03.871745 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.872349 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.873639 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.874221 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.874340 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.874381 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.874556 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.874638 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 07:02:03.874792 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.870506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:45533 (local address 127.11.106.65:48845)
0619 07:02:03.871052 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:03.871062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:03.871189 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:03.871488 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:03.871490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:03.871509 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:03.871612 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.871617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.872472 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.872476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.873506 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:03.873517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.874340 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.874344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:03.874430 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.874451 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 07:02:03.874522 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 07:02:03.874616 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":278,"spinlock_wait_cycles":24704,"thread_start_us":100,"threads_started":1}
I20260619 07:02:03.875289 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d70e873ed643a8b89cb6906f7b5e4d" candidate_uuid: "7c8d387afa474394bfaf0344084dbea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" is_pre_election: true
I20260619 07:02:03.875391 30629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d70e873ed643a8b89cb6906f7b5e4d" candidate_uuid: "7c8d387afa474394bfaf0344084dbea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" is_pre_election: true
I20260619 07:02:03.875630 30760 raft_consensus.cc:2454] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8d387afa474394bfaf0344084dbea5 in term 0.
I20260619 07:02:03.875630 30629 raft_consensus.cc:2454] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c8d387afa474394bfaf0344084dbea5 in term 0.
I20260619 07:02:03.876159 30432 leader_election.cc:302] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 68a9953aaf1a4ec39f9f516d553aef2b, 68caa0156f8c4f65b2aabf07b8a9b1e4, 7c8d387afa474394bfaf0344084dbea5; no voters:
I20260619 07:02:03.876449 30831 raft_consensus.cc:2781] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:02:03.876555 30831 raft_consensus.cc:482] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:02:03.876613 30831 raft_consensus.cc:3037] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:03.877621 30831 raft_consensus.cc:504] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } }
I20260619 07:02:03.877942 30831 leader_election.cc:288] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [CANDIDATE]: Term 1 election: Requested vote from peers 68a9953aaf1a4ec39f9f516d553aef2b (127.11.106.67:38523), 68caa0156f8c4f65b2aabf07b8a9b1e4 (127.11.106.66:45533)
I20260619 07:02:03.878281 30629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d70e873ed643a8b89cb6906f7b5e4d" candidate_uuid: "7c8d387afa474394bfaf0344084dbea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4"
I20260619 07:02:03.878274 30760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18d70e873ed643a8b89cb6906f7b5e4d" candidate_uuid: "7c8d387afa474394bfaf0344084dbea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a9953aaf1a4ec39f9f516d553aef2b"
I20260619 07:02:03.878417 30629 raft_consensus.cc:3037] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:03.878413 30760 raft_consensus.cc:3037] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:03.879423 30629 raft_consensus.cc:2454] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8d387afa474394bfaf0344084dbea5 in term 1.
I20260619 07:02:03.879482 30760 raft_consensus.cc:2454] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c8d387afa474394bfaf0344084dbea5 in term 1.
I20260619 07:02:03.879732 30432 leader_election.cc:302] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68caa0156f8c4f65b2aabf07b8a9b1e4, 7c8d387afa474394bfaf0344084dbea5; no voters:
I20260619 07:02:03.879904 30831 raft_consensus.cc:2781] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:03.880127 30831 raft_consensus.cc:686] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [term 1 LEADER]: Becoming Leader. State: Replica: 7c8d387afa474394bfaf0344084dbea5, State: Running, Role: LEADER
I20260619 07:02:03.880457 30831 consensus_queue.cc:237] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } }
I20260619 07:02:03.883688 30355 catalog_manager.cc:5495] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c8d387afa474394bfaf0344084dbea5 (127.11.106.65). New cstate: current_term: 1 leader_uuid: "7c8d387afa474394bfaf0344084dbea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c8d387afa474394bfaf0344084dbea5" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 37163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 } health_report { overall_health: UNKNOWN } } }
I20260619 07:02:03.904316 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.901089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.11.106.65:37163)
0619 07:02:03.901221 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:03.901224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:03.901238 (+ 14us) server_negotiation.cc:407] Connection header received
0619 07:02:03.901344 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:03.901346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:03.901403 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:03.901484 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:03.901698 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.902173 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.902968 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:03.903168 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:03.903240 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:03.903493 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:03.903897 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:03.904005 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:03.904096 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 07:02:03.904126 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 07:02:03.910437 30629 raft_consensus.cc:1261] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [term 1 FOLLOWER]: Refusing update from remote peer 7c8d387afa474394bfaf0344084dbea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:03.910437 30760 raft_consensus.cc:1261] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [term 1 FOLLOWER]: Refusing update from remote peer 7c8d387afa474394bfaf0344084dbea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:03.911175 30831 consensus_queue.cc:1035] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a9953aaf1a4ec39f9f516d553aef2b" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:03.911389 30833 consensus_queue.cc:1035] T 18d70e873ed643a8b89cb6906f7b5e4d P 7c8d387afa474394bfaf0344084dbea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68caa0156f8c4f65b2aabf07b8a9b1e4" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 45533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:03.920430 30837 mvcc.cc:205] Tried to move back new op lower bound from 7298467937929527296 to 7298467937817034752. Current Snapshot: MvccSnapshot[applied={T|T < 7298467937929527296 or (T in {7298467937929527296})}]
I20260619 07:02:03.922676 30838 mvcc.cc:205] Tried to move back new op lower bound from 7298467937929527296 to 7298467937817034752. Current Snapshot: MvccSnapshot[applied={T|T < 7298467937929527296}]
I20260619 07:02:03.927341 30355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:03.927515 30355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:03.930007 30355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 647509b01c7a4ec395ab3c22bb7765b2: Sending DeleteTablet for 3 replicas of tablet 18d70e873ed643a8b89cb6906f7b5e4d
I20260619 07:02:03.930840 30478 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d70e873ed643a8b89cb6906f7b5e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:03 UTC) from {username='slave'} at 127.0.0.1:36656
I20260619 07:02:03.931010 30740 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d70e873ed643a8b89cb6906f7b5e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:03 UTC) from {username='slave'} at 127.0.0.1:36796
I20260619 07:02:03.931010 30609 tablet_service.cc:1514] Processing DeleteTablet for tablet 18d70e873ed643a8b89cb6906f7b5e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:03 UTC) from {username='slave'} at 127.0.0.1:49280
I20260619 07:02:03.931499 30847 tablet_replica.cc:330] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: stopping tablet replica
I20260619 07:02:03.931821 30847 raft_consensus.cc:2229] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:02:03.932155 30847 raft_consensus.cc:2258] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:03.932256 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 30416
I20260619 07:02:03.933977 30848 tablet_replica.cc:330] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: stopping tablet replica
I20260619 07:02:03.934171 30847 ts_tablet_manager.cc:1905] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:03.934289 30848 raft_consensus.cc:2229] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:02:03.934576 30848 raft_consensus.cc:2258] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:03.936895 30848 ts_tablet_manager.cc:1905] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:03.937397 30847 ts_tablet_manager.cc:1918] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:03.937489 30847 log.cc:1196] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/18d70e873ed643a8b89cb6906f7b5e4d
I20260619 07:02:03.937784 30847 ts_tablet_manager.cc:1939] T 18d70e873ed643a8b89cb6906f7b5e4d P 68caa0156f8c4f65b2aabf07b8a9b1e4: Deleting consensus metadata
I20260619 07:02:03.939051 30341 catalog_manager.cc:4841] TS 68caa0156f8c4f65b2aabf07b8a9b1e4 (127.11.106.66:45533): tablet 18d70e873ed643a8b89cb6906f7b5e4d (table test-table [id=8e21a8af9dd6488daf55ca1bd7fe2999]) successfully deleted
W20260619 07:02:03.939544 30340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:37163 (error 108)
I20260619 07:02:03.939692 30848 ts_tablet_manager.cc:1918] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:03.939719 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 30547
I20260619 07:02:03.940449 30848 log.cc:1196] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/18d70e873ed643a8b89cb6906f7b5e4d
I20260619 07:02:03.940780 30848 ts_tablet_manager.cc:1939] T 18d70e873ed643a8b89cb6906f7b5e4d P 68a9953aaf1a4ec39f9f516d553aef2b: Deleting consensus metadata
I20260619 07:02:03.941383 30819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:03.941045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:37163 (local address 127.0.0.1:36668)
0619 07:02:03.941200 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:03.941279 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 07:02:03.941529 30340 catalog_manager.cc:4584] TS 7c8d387afa474394bfaf0344084dbea5 (127.11.106.65:37163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18d70e873ed643a8b89cb6906f7b5e4d: Network error: Client connection negotiation failed: client connection to 127.11.106.65:37163: connect: Connection refused (error 111)
I20260619 07:02:03.941982 30340 catalog_manager.cc:4841] TS 68a9953aaf1a4ec39f9f516d553aef2b (127.11.106.67:38523): tablet 18d70e873ed643a8b89cb6906f7b5e4d (table test-table [id=8e21a8af9dd6488daf55ca1bd7fe2999]) successfully deleted
I20260619 07:02:03.946923 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 30678
I20260619 07:02:03.952385 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 30325
2026-06-19T07:02:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1021 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:04 dist-test-slave-10d0 krb5kdc[30852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:04 dist-test-slave-10d0 krb5kdc[30852](info): set up 2 sockets
Jun 19 07:02:04 dist-test-slave-10d0 krb5kdc[30852](info): commencing operation
krb5kdc: starting...
W20260619 07:02:04.997212 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:05 dist-test-slave-10d0 krb5kdc[30852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:05.121302 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:33085
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:33085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:05.232106 30865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:05.232406 30865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:05.232465 30865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:05.236433 30865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:05.236517 30865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:05.236570 30865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:05.236631 30865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:05.236667 30865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:05.241915 30865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:33085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:33085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:05.242900 30865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:05.243938 30865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:05.252393 30873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:05.252408 30871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:05.252410 30870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:05.252626 30865 server_base.cc:1034] running on GCE node
I20260619 07:02:05.253517 30865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:05.254729 30865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:05.255956 30865 hybrid_clock.cc:648] HybridClock initialized: now 1781852525255939 us; error 37 us; skew 500 ppm
Jun 19 07:02:05 dist-test-slave-10d0 krb5kdc[30852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852525, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:05.259047 30865 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:05.259943 30865 webserver.cc:460] Webserver started at http://127.11.106.126:43441/ using document root <none> and password file <none>
I20260619 07:02:05.260507 30865 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:05.260658 30865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:05.260891 30865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:05.262876 30865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "ed846ab761984620996df083c946bdd6"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "52c3156c935f1ea6417265fda482f541"
server_key_iv: "5f4d9b8b06310ff5ae129fd106b1d118"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.263440 30865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "ed846ab761984620996df083c946bdd6"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "52c3156c935f1ea6417265fda482f541"
server_key_iv: "5f4d9b8b06310ff5ae129fd106b1d118"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.267046 30865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 07:02:05.269362 30880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.270553 30865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:05.270701 30865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "ed846ab761984620996df083c946bdd6"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "52c3156c935f1ea6417265fda482f541"
server_key_iv: "5f4d9b8b06310ff5ae129fd106b1d118"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.270810 30865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:05.282339 30865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:05.285208 30865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:05.285410 30865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:05.292661 30865 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:33085
I20260619 07:02:05.292699 30932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:33085 every 8 connection(s)
I20260619 07:02:05.293646 30865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:05.296285 30933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:05.297303 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 30865
I20260619 07:02:05.297441 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:05.297734 11689 external_mini_cluster.cc:1428] Setting key 78e93f46b975348c6b584fd78ea8df6b
I20260619 07:02:05.302240 30933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6: Bootstrap starting.
I20260619 07:02:05.304422 30933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:05.305326 30933 log.cc:824] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6: Log is configured to *not* fsync() on all Append() calls
Jun 19 07:02:05 dist-test-slave-10d0 krb5kdc[30852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:05.307723 30933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6: No bootstrap required, opened a new log
I20260619 07:02:05.311681 30933 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed846ab761984620996df083c946bdd6" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33085 } }
I20260619 07:02:05.311906 30933 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:05.311962 30933 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed846ab761984620996df083c946bdd6, State: Initialized, Role: FOLLOWER
I20260619 07:02:05.312444 30933 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed846ab761984620996df083c946bdd6" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33085 } }
I20260619 07:02:05.312641 30933 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:05.312716 30933 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:05.312832 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:05.299186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.11.106.126:33085)
0619 07:02:05.299641 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:05.299656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:05.299693 (+ 37us) server_negotiation.cc:407] Connection header received
0619 07:02:05.300329 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:05.300346 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:05.300773 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:05.301106 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:05.301913 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.302923 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.303768 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.304242 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.304410 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:05.307554 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:05.307560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:05.307574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:05.307611 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:05.309563 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:05.310090 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:05.310095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:05.310100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:05.310216 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:05.310509 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:05.310511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:05.310513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:05.310925 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:05.311104 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:05.311408 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 07:02:05.311784 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":133,"threads_started":1}
I20260619 07:02:05.312856 30933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:05.313820 30933 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed846ab761984620996df083c946bdd6" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33085 } }
I20260619 07:02:05.314165 30933 leader_election.cc:302] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed846ab761984620996df083c946bdd6; no voters:
I20260619 07:02:05.314493 30933 leader_election.cc:288] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:05.314611 30938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:05.314836 30938 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [term 1 LEADER]: Becoming Leader. State: Replica: ed846ab761984620996df083c946bdd6, State: Running, Role: LEADER
I20260619 07:02:05.315227 30938 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed846ab761984620996df083c946bdd6" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33085 } }
I20260619 07:02:05.315809 30933 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:05.316493 30938 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed846ab761984620996df083c946bdd6. Latest consensus state: current_term: 1 leader_uuid: "ed846ab761984620996df083c946bdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed846ab761984620996df083c946bdd6" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33085 } } }
I20260619 07:02:05.316754 30938 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:05.316954 30939 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed846ab761984620996df083c946bdd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed846ab761984620996df083c946bdd6" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33085 } } }
I20260619 07:02:05.317147 30939 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:05.317237 30947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:05.320693 30947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:05.325994 30947 catalog_manager.cc:1347] Generated new cluster ID: 6f2002f268d34bf8b6c2714abe0cc976
I20260619 07:02:05.326062 30947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:05.356943 30947 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:05.357859 30947 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:05.363441 30947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6: Generated new TSK 0
I20260619 07:02:05.364144 30947 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.106.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
Entry for principal kudu/127.11.106.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab.
I20260619 07:02:05.393378 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33085
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:05.503973 30959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:05.504221 30959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:05.504263 30959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:05.508280 30959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:05.508338 30959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:05.508395 30959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:02:05.513820 30959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.65.keytab
--principal=kudu/127.11.106.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:05.514880 30959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:05.515918 30959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:05.525825 30965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:05.525815 30967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:05.525815 30964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:05.526016 30959 server_base.cc:1034] running on GCE node
I20260619 07:02:05.526703 30959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:05.527436 30959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:05.528627 30959 hybrid_clock.cc:648] HybridClock initialized: now 1781852525528613 us; error 36 us; skew 500 ppm
Jun 19 07:02:05 dist-test-slave-10d0 krb5kdc[30852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852525, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:05.531286 30959 init.cc:377] Logged in from keytab as kudu/127.11.106.65@KRBTEST.COM (short username kudu)
I20260619 07:02:05.532220 30959 webserver.cc:460] Webserver started at http://127.11.106.65:41945/ using document root <none> and password file <none>
I20260619 07:02:05.532917 30959 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:05.533008 30959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:05.533248 30959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:05.535300 30959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "fe0d41299fea4411bd5ad84e4b436e03"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "8414b6e95d8d5255393efeefb3d8b8b1"
server_key_iv: "a2a7bfbf41231763d3048de280e0f011"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.535867 30959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "fe0d41299fea4411bd5ad84e4b436e03"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "8414b6e95d8d5255393efeefb3d8b8b1"
server_key_iv: "a2a7bfbf41231763d3048de280e0f011"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.539680 30959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 07:02:05.542150 30974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.543474 30959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:05.543612 30959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "fe0d41299fea4411bd5ad84e4b436e03"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "8414b6e95d8d5255393efeefb3d8b8b1"
server_key_iv: "a2a7bfbf41231763d3048de280e0f011"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.543725 30959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:05.559262 30959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:05.562732 30959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:05.562935 30959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:05.563556 30959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:05.564428 30959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:05.564502 30959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.564604 30959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:05.564656 30959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.574280 30959 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:34035
I20260619 07:02:05.574297 31087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:34035 every 8 connection(s)
I20260619 07:02:05.575287 30959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:02:05.579496 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 30959
I20260619 07:02:05.579661 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 07:02:05.579983 11689 external_mini_cluster.cc:1428] Setting key ae3e9cc377a7787f1314d4c599f2929b
Jun 19 07:02:05 dist-test-slave-10d0 krb5kdc[30852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852525, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.65@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.66@KRBTEST.COM; defaulting to no policy
I20260619 07:02:05.590107 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:05.577200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:57703 (local address 127.11.106.126:33085)
0619 07:02:05.577342 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:05.577346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:05.578206 (+ 860us) server_negotiation.cc:407] Connection header received
0619 07:02:05.579552 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:05.579555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:05.579612 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:05.579726 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:05.581030 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.581554 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.582708 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.583121 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.583241 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:05.585815 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:05.585822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:05.585826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:05.585858 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:05.587796 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:05.588366 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:05.588370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:05.588372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:05.588464 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:05.588895 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:05.588899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:05.588900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:05.589192 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:05.589304 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:05.589853 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 07:02:05.589977 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 07:02:05.590979 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:05.577598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:33085 (local address 127.11.106.65:57703)
0619 07:02:05.578045 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:05.578088 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:05.579331 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:05.579906 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:05.579913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:05.580326 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:05.580885 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:05.580899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.581689 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.581697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:05.582571 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:05.582580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.583308 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.583312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:05.583462 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:05.584103 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:05.584120 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:05.585629 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:05.587933 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:05.587938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:05.587950 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:05.588249 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:05.588639 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:05.588642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:05.588643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:05.588786 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:05.589300 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:05.589305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:05.589546 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 07:02:05.589770 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 07:02:05.590038 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.66@KRBTEST.COM" created.
I20260619 07:02:05.592484 31088 heartbeater.cc:347] Connected to a master server at 127.11.106.126:33085
I20260619 07:02:05.592756 31088 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:05.593403 31088 heartbeater.cc:511] Master 127.11.106.126:33085 requested a full tablet report, sending...
I20260619 07:02:05.595099 30897 ts_manager.cc:194] Registered new tserver with Master: fe0d41299fea4411bd5ad84e4b436e03 (127.11.106.65:34035)
I20260619 07:02:05.596968 30897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.65@KRBTEST.COM'} at 127.11.106.65:57703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
Entry for principal kudu/127.11.106.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab.
I20260619 07:02:05.609542 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33085
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:05.719108 31093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:05.719398 31093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:05.719439 31093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:05.723681 31093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:05.723744 31093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:05.723803 31093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:02:05.729137 31093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.66.keytab
--principal=kudu/127.11.106.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:05.730100 31093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:05.731060 31093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:05.740958 31098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:05.740976 31101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:05.740965 31099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:05.741530 31093 server_base.cc:1034] running on GCE node
I20260619 07:02:05.741958 31093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:05.742653 31093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:05.743837 31093 hybrid_clock.cc:648] HybridClock initialized: now 1781852525743809 us; error 53 us; skew 500 ppm
Jun 19 07:02:05 dist-test-slave-10d0 krb5kdc[30852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852525, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:05.746701 31093 init.cc:377] Logged in from keytab as kudu/127.11.106.66@KRBTEST.COM (short username kudu)
I20260619 07:02:05.747664 31093 webserver.cc:460] Webserver started at http://127.11.106.66:43621/ using document root <none> and password file <none>
I20260619 07:02:05.748276 31093 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:05.748379 31093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:05.748683 31093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:05.750756 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "4f091d2f4e2348848c4312cde543dd92"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "6738bb8c606e0884a6c773e9dcdefb50"
server_key_iv: "b2d9b9a4e89db54bf6dc7ec1460aa364"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.751324 31093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "4f091d2f4e2348848c4312cde543dd92"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "6738bb8c606e0884a6c773e9dcdefb50"
server_key_iv: "b2d9b9a4e89db54bf6dc7ec1460aa364"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.756179 31093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 07:02:05.758766 31108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.760008 31093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:05.760150 31093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "4f091d2f4e2348848c4312cde543dd92"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "6738bb8c606e0884a6c773e9dcdefb50"
server_key_iv: "b2d9b9a4e89db54bf6dc7ec1460aa364"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.760272 31093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:05.773286 31093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:05.776507 31093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:05.776803 31093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:05.777390 31093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:05.778195 31093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:05.778241 31093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.778280 31093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:05.778295 31093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.787400 31093 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:37571
I20260619 07:02:05.787420 31221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:37571 every 8 connection(s)
I20260619 07:02:05.788318 31093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:02:05.795683 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 31093
I20260619 07:02:05.795820 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 07:02:05.796100 11689 external_mini_cluster.cc:1428] Setting key 4d1291a64a4422ae8ced59c3f6f4d17a
Jun 19 07:02:05 dist-test-slave-10d0 krb5kdc[30852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852525, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.66@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:05.802248 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:05.790235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:56973 (local address 127.11.106.126:33085)
0619 07:02:05.790389 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:05.790392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:05.791262 (+ 870us) server_negotiation.cc:407] Connection header received
0619 07:02:05.792626 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:05.792629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:05.792691 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:05.792798 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:05.793956 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.794478 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.795621 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.795943 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.796055 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:05.798516 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:05.798520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:05.798522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:05.798547 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:05.800009 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:05.800645 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:05.800648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:05.800649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:05.800723 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:05.801120 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:05.801122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:05.801123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:05.801281 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:05.801353 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:05.802030 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 07:02:05.802130 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 07:02:05.803508 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:05.790644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:33085 (local address 127.11.106.66:56973)
0619 07:02:05.791100 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:05.791152 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:05.792370 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:05.792977 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:05.792983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:05.793342 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:05.793813 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:05.793826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.794603 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.794609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:05.795486 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:05.795495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:05.796093 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:05.796097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:05.796318 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:05.796940 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:05.796956 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:05.798357 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:05.800133 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:05.800142 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:05.800160 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:05.800524 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:05.800817 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:05.800823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:05.800828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:05.801017 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:05.801391 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:05.801400 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:05.801758 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 07:02:05.801976 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 07:02:05.802341 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260619 07:02:05.804749 31222 heartbeater.cc:347] Connected to a master server at 127.11.106.126:33085
I20260619 07:02:05.804991 31222 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:05.805461 31222 heartbeater.cc:511] Master 127.11.106.126:33085 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.106.67@KRBTEST.COM; defaulting to no policy
I20260619 07:02:05.806506 30897 ts_manager.cc:194] Registered new tserver with Master: 4f091d2f4e2348848c4312cde543dd92 (127.11.106.66:37571)
I20260619 07:02:05.807082 30897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.66@KRBTEST.COM'} at 127.11.106.66:56973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
Entry for principal kudu/127.11.106.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab.
I20260619 07:02:05.824220 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33085
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:05.934146 31227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:05.934407 31227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:05.934487 31227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:05.938488 31227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:05.938560 31227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:05.938644 31227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:02:05.943707 31227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.67.keytab
--principal=kudu/127.11.106.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:33085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:05.944759 31227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:05.945780 31227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:05.955622 31232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:05.955708 31227 server_base.cc:1034] running on GCE node
W20260619 07:02:05.955622 31235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:05.955641 31233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:05.956300 31227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:05.956943 31227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:05.958138 31227 hybrid_clock.cc:648] HybridClock initialized: now 1781852525958113 us; error 27 us; skew 500 ppm
Jun 19 07:02:05 dist-test-slave-10d0 krb5kdc[30852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852525, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:05.960785 31227 init.cc:377] Logged in from keytab as kudu/127.11.106.67@KRBTEST.COM (short username kudu)
I20260619 07:02:05.961560 31227 webserver.cc:460] Webserver started at http://127.11.106.67:43627/ using document root <none> and password file <none>
I20260619 07:02:05.962062 31227 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:05.962127 31227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:05.962291 31227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:05.964339 31227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "a2af32d6018a48559428acc6675f1171"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "a9b79eca34dc38e76df9ecac356cbbca"
server_key_iv: "52a791a47cf02eaa7c74b11c8c9cb5dc"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.964936 31227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "a2af32d6018a48559428acc6675f1171"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "a9b79eca34dc38e76df9ecac356cbbca"
server_key_iv: "52a791a47cf02eaa7c74b11c8c9cb5dc"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.968475 31227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 07:02:05.970820 31242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.972019 31227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 07:02:05.972121 31227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "a2af32d6018a48559428acc6675f1171"
format_stamp: "Formatted at 2026-06-19 07:02:05 on dist-test-slave-10d0"
server_key: "a9b79eca34dc38e76df9ecac356cbbca"
server_key_iv: "52a791a47cf02eaa7c74b11c8c9cb5dc"
server_key_version: "encryptionkey@0"
I20260619 07:02:05.972203 31227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:05.985647 31227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:05.988471 31227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:05.988679 31227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:05.989210 31227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:05.989957 31227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:05.990002 31227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.990041 31227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:05.990056 31227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:05.999459 31227 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:46605
I20260619 07:02:05.999478 31355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:46605 every 8 connection(s)
I20260619 07:02:06.000355 31227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
Jun 19 07:02:06 dist-test-slave-10d0 krb5kdc[30852](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852525, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.67@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:06.010131 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 31227
I20260619 07:02:06.010289 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 07:02:06.010624 11689 external_mini_cluster.cc:1428] Setting key 839db4e01ef612cd47d3c6861f4691e0
I20260619 07:02:06.014303 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.002253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:58859 (local address 127.11.106.126:33085)
0619 07:02:06.002432 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.002438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.003300 (+ 862us) server_negotiation.cc:407] Connection header received
0619 07:02:06.004653 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.004655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.004705 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.004769 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:06.005867 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.006389 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.007625 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.007839 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.007920 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.010515 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:06.010518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:06.010520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:06.010545 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:06.012312 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:06.012906 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:06.012909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:06.012911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:06.013003 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:06.013348 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:06.013350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:06.013351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:06.013527 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:06.013617 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.014086 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.014187 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 07:02:06.015098 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.002673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:33085 (local address 127.11.106.67:58859)
0619 07:02:06.003135 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:06.003179 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:06.004382 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:06.004904 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:06.004911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:06.005257 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:06.005731 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.005743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.006641 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.006644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.007509 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.007518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.007971 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.007973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.008113 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.008694 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:06.008710 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:06.010282 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:06.012421 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:06.012426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:06.012439 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:06.012781 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:06.013105 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:06.013108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:06.013110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:06.013254 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:06.013635 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:06.013640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:06.013872 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 07:02:06.014022 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 07:02:06.014274 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260619 07:02:06.016232 31356 heartbeater.cc:347] Connected to a master server at 127.11.106.126:33085
I20260619 07:02:06.016474 31356 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:06.017063 31356 heartbeater.cc:511] Master 127.11.106.126:33085 requested a full tablet report, sending...
I20260619 07:02:06.018082 30897 ts_manager.cc:194] Registered new tserver with Master: a2af32d6018a48559428acc6675f1171 (127.11.106.67:46605)
I20260619 07:02:06.018657 30897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.106.67@KRBTEST.COM'} at 127.11.106.67:58859
I20260619 07:02:06.024765 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:02:06.033747 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.026657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39606 (local address 127.11.106.126:33085)
0619 07:02:06.026792 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.026795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.027025 (+ 230us) server_negotiation.cc:407] Connection header received
0619 07:02:06.027130 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.027132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.027171 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.027239 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:06.027934 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.028438 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.029279 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.029484 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.029562 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.030847 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:06.030850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:06.030852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:06.030873 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:06.032097 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:06.032467 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:06.032469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:06.032471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:06.032588 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:06.032928 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:06.032930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:06.032931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:06.033136 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:06.033244 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.033442 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.033533 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 07:02:06.036412 30897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:02:06.039645 30897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:02:06.054857 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.048595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:37571 (local address 127.0.0.1:48140)
0619 07:02:06.049615 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:06.049698 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:06.049839 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:06.050454 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:06.050460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:06.050483 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:06.050713 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.050722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.052476 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.052478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.053367 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.053377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.054295 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.054298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.054406 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.054446 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 07:02:06.054563 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 07:02:06.054689 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":17,"thread_start_us":79,"threads_started":1}
I20260619 07:02:06.054893 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.048761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.11.106.66:37571)
0619 07:02:06.049572 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.049585 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.049987 (+ 402us) server_negotiation.cc:407] Connection header received
0619 07:02:06.050076 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.050082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.050294 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.050455 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:06.050833 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.052078 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.053500 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.054171 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.054285 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.054433 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.054618 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.054723 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":259,"threads_started":1}
I20260619 07:02:06.055676 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.048401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.11.106.67:46605)
0619 07:02:06.049719 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.049736 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.050042 (+ 306us) server_negotiation.cc:407] Connection header received
0619 07:02:06.050284 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.050290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.050490 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.050635 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:06.050883 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.052078 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.054036 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.054830 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.054935 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.055096 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.055411 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.055513 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":836,"threads_started":1}
I20260619 07:02:06.056123 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.049359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:46605 (local address 127.0.0.1:34992)
0619 07:02:06.049942 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:06.049957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:06.050107 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:06.050656 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:06.050659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:06.050677 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:06.050782 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.050789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.052225 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.052231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.053883 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.053894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.055158 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.055161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.055246 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.055272 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 07:02:06.055869 (+ 597us) client_negotiation.cc:240] Negotiation successful
0619 07:02:06.055978 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":294,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260619 07:02:06.056298 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.049874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36724 (local address 127.11.106.65:34035)
0619 07:02:06.050260 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.050275 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.050295 (+ 20us) server_negotiation.cc:407] Connection header received
0619 07:02:06.050370 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.050376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.050555 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.050701 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:06.051064 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.052142 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.054021 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.055497 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.055618 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.055741 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.056062 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.056165 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":95,"threads_started":1}
I20260619 07:02:06.056233 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.049704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:34035 (local address 127.0.0.1:36724)
0619 07:02:06.050000 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:06.050013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:06.050097 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:06.050848 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:06.050853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:06.050874 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:06.050948 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.050952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.052282 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.052284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.053883 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.053894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.055613 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.055617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.055708 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.055928 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 07:02:06.056006 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 07:02:06.056064 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":182,"thread_start_us":90,"threads_started":1}
I20260619 07:02:06.058517 31156 tablet_service.cc:1467] Processing CreateTablet for tablet 7c78c611183943ec879a4aadcd51ac52 (DEFAULT_TABLE table=test-table [id=4345f47d2a1b48ba991cd6862ddc9442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:06.058794 31022 tablet_service.cc:1467] Processing CreateTablet for tablet 7c78c611183943ec879a4aadcd51ac52 (DEFAULT_TABLE table=test-table [id=4345f47d2a1b48ba991cd6862ddc9442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:06.059069 31289 tablet_service.cc:1467] Processing CreateTablet for tablet 7c78c611183943ec879a4aadcd51ac52 (DEFAULT_TABLE table=test-table [id=4345f47d2a1b48ba991cd6862ddc9442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:06.059729 31156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c78c611183943ec879a4aadcd51ac52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:06.059767 31022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c78c611183943ec879a4aadcd51ac52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:06.059963 31289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c78c611183943ec879a4aadcd51ac52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:06.066043 31373 tablet_bootstrap.cc:492] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03: Bootstrap starting.
I20260619 07:02:06.067353 31375 tablet_bootstrap.cc:492] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: Bootstrap starting.
I20260619 07:02:06.067353 31374 tablet_bootstrap.cc:492] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: Bootstrap starting.
I20260619 07:02:06.068250 31373 tablet_bootstrap.cc:654] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:06.069496 31375 tablet_bootstrap.cc:654] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:06.069496 31374 tablet_bootstrap.cc:654] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:06.069620 31373 log.cc:824] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:06.070794 31374 log.cc:824] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:06.070794 31375 log.cc:824] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:06.071581 31373 tablet_bootstrap.cc:492] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03: No bootstrap required, opened a new log
I20260619 07:02:06.071789 31373 ts_tablet_manager.cc:1397] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 07:02:06.072497 31375 tablet_bootstrap.cc:492] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: No bootstrap required, opened a new log
I20260619 07:02:06.072718 31374 tablet_bootstrap.cc:492] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: No bootstrap required, opened a new log
I20260619 07:02:06.072737 31375 ts_tablet_manager.cc:1397] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260619 07:02:06.072899 31374 ts_tablet_manager.cc:1397] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 07:02:06.075730 31373 raft_consensus.cc:348] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } }
I20260619 07:02:06.075696 31375 raft_consensus.cc:348] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } }
I20260619 07:02:06.076009 31375 raft_consensus.cc:374] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:06.076099 31375 raft_consensus.cc:729] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2af32d6018a48559428acc6675f1171, State: Initialized, Role: FOLLOWER
I20260619 07:02:06.076288 31373 raft_consensus.cc:374] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:06.076402 31373 raft_consensus.cc:729] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0d41299fea4411bd5ad84e4b436e03, State: Initialized, Role: FOLLOWER
I20260619 07:02:06.076596 31375 consensus_queue.cc:260] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } }
I20260619 07:02:06.076912 31373 consensus_queue.cc:260] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } }
I20260619 07:02:06.077332 31356 heartbeater.cc:503] Master 127.11.106.126:33085 was elected leader, sending a full tablet report...
I20260619 07:02:06.077479 31374 raft_consensus.cc:348] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } }
I20260619 07:02:06.077721 31374 raft_consensus.cc:374] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:06.077811 31374 raft_consensus.cc:729] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f091d2f4e2348848c4312cde543dd92, State: Initialized, Role: FOLLOWER
I20260619 07:02:06.077979 31373 ts_tablet_manager.cc:1428] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 07:02:06.078348 31088 heartbeater.cc:503] Master 127.11.106.126:33085 was elected leader, sending a full tablet report...
I20260619 07:02:06.078291 31374 consensus_queue.cc:260] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } }
I20260619 07:02:06.079026 31375 ts_tablet_manager.cc:1428] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260619 07:02:06.079205 31374 ts_tablet_manager.cc:1428] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 07:02:06.079523 31222 heartbeater.cc:503] Master 127.11.106.126:33085 was elected leader, sending a full tablet report...
W20260619 07:02:06.251449 31357 tablet.cc:2369] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 07:02:06.289505 31223 tablet.cc:2369] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 07:02:06.326658 31089 tablet.cc:2369] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:02:06.360193 31380 raft_consensus.cc:482] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:02:06.360409 31380 raft_consensus.cc:504] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } }
I20260619 07:02:06.361630 31380 leader_election.cc:288] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2af32d6018a48559428acc6675f1171 (127.11.106.67:46605), 4f091d2f4e2348848c4312cde543dd92 (127.11.106.66:37571)
I20260619 07:02:06.367158 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.362017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:57405 (local address 127.11.106.66:37571)
0619 07:02:06.362158 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.362163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.362383 (+ 220us) server_negotiation.cc:407] Connection header received
0619 07:02:06.362527 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.362529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.362582 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.362681 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:06.363123 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.363899 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.365811 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.366397 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.366527 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.366577 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.366948 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.367034 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 07:02:06.367172 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.362036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:43691 (local address 127.11.106.67:46605)
0619 07:02:06.362276 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.362280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.362381 (+ 101us) server_negotiation.cc:407] Connection header received
0619 07:02:06.362526 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.362529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.362582 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.362679 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:06.363120 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.363924 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.365811 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.366402 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.366526 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.366577 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.366948 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.367028 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260619 07:02:06.367223 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.361897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:46605 (local address 127.11.106.65:43691)
0619 07:02:06.362289 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:06.362304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:06.362420 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:06.362704 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:06.362707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:06.362732 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:06.362991 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.362999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.364032 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.364035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.365676 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.365687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.366535 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.366539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.366762 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.366804 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 07:02:06.366889 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 07:02:06.367026 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":383,"spinlock_wait_cycles":17792,"thread_start_us":103,"threads_started":1}
I20260619 07:02:06.367226 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.361897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:37571 (local address 127.11.106.65:57405)
0619 07:02:06.362262 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:06.362276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:06.362420 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:06.362704 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:06.362707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:06.362732 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:06.362994 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.362999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.364032 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.364035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.365676 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.365687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.366535 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.366539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.366759 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.366811 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 07:02:06.366889 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 07:02:06.367025 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":351,"spinlock_wait_cycles":24960,"thread_start_us":98,"threads_started":1}
I20260619 07:02:06.368014 31310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c78c611183943ec879a4aadcd51ac52" candidate_uuid: "fe0d41299fea4411bd5ad84e4b436e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2af32d6018a48559428acc6675f1171" is_pre_election: true
I20260619 07:02:06.368016 31176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c78c611183943ec879a4aadcd51ac52" candidate_uuid: "fe0d41299fea4411bd5ad84e4b436e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f091d2f4e2348848c4312cde543dd92" is_pre_election: true
I20260619 07:02:06.368338 31176 raft_consensus.cc:2454] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe0d41299fea4411bd5ad84e4b436e03 in term 0.
I20260619 07:02:06.368338 31310 raft_consensus.cc:2454] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe0d41299fea4411bd5ad84e4b436e03 in term 0.
I20260619 07:02:06.368908 30977 leader_election.cc:302] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4f091d2f4e2348848c4312cde543dd92, a2af32d6018a48559428acc6675f1171, fe0d41299fea4411bd5ad84e4b436e03; no voters:
I20260619 07:02:06.369264 31380 raft_consensus.cc:2781] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:02:06.369386 31380 raft_consensus.cc:482] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:02:06.369452 31380 raft_consensus.cc:3037] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:06.370513 31380 raft_consensus.cc:504] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } }
I20260619 07:02:06.370879 31380 leader_election.cc:288] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [CANDIDATE]: Term 1 election: Requested vote from peers a2af32d6018a48559428acc6675f1171 (127.11.106.67:46605), 4f091d2f4e2348848c4312cde543dd92 (127.11.106.66:37571)
I20260619 07:02:06.371269 31176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c78c611183943ec879a4aadcd51ac52" candidate_uuid: "fe0d41299fea4411bd5ad84e4b436e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f091d2f4e2348848c4312cde543dd92"
I20260619 07:02:06.371381 31176 raft_consensus.cc:3037] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:06.372432 31176 raft_consensus.cc:2454] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0d41299fea4411bd5ad84e4b436e03 in term 1.
I20260619 07:02:06.372679 31310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c78c611183943ec879a4aadcd51ac52" candidate_uuid: "fe0d41299fea4411bd5ad84e4b436e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2af32d6018a48559428acc6675f1171"
I20260619 07:02:06.372800 30977 leader_election.cc:302] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f091d2f4e2348848c4312cde543dd92, fe0d41299fea4411bd5ad84e4b436e03; no voters:
I20260619 07:02:06.372830 31310 raft_consensus.cc:3037] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:06.372983 31380 raft_consensus.cc:2781] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:06.373253 31380 raft_consensus.cc:686] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [term 1 LEADER]: Becoming Leader. State: Replica: fe0d41299fea4411bd5ad84e4b436e03, State: Running, Role: LEADER
I20260619 07:02:06.373596 31380 consensus_queue.cc:237] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } }
I20260619 07:02:06.374142 31310 raft_consensus.cc:2454] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe0d41299fea4411bd5ad84e4b436e03 in term 1.
I20260619 07:02:06.377128 30895 catalog_manager.cc:5495] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe0d41299fea4411bd5ad84e4b436e03 (127.11.106.65). New cstate: current_term: 1 leader_uuid: "fe0d41299fea4411bd5ad84e4b436e03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe0d41299fea4411bd5ad84e4b436e03" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 34035 } health_report { overall_health: HEALTHY } } }
I20260619 07:02:06.417508 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.413242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.11.106.65:34035)
0619 07:02:06.413373 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.413378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.413393 (+ 15us) server_negotiation.cc:407] Connection header received
0619 07:02:06.413642 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.413645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.413711 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.413833 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:06.414112 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.414665 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.415665 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.415992 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.416114 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.416487 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:06.417106 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:06.417250 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.417295 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.417357 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 07:02:06.423465 31310 raft_consensus.cc:1261] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [term 1 FOLLOWER]: Refusing update from remote peer fe0d41299fea4411bd5ad84e4b436e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:06.423465 31176 raft_consensus.cc:1261] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [term 1 FOLLOWER]: Refusing update from remote peer fe0d41299fea4411bd5ad84e4b436e03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:06.424192 31380 consensus_queue.cc:1035] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2af32d6018a48559428acc6675f1171" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:06.424436 31384 consensus_queue.cc:1035] T 7c78c611183943ec879a4aadcd51ac52 P fe0d41299fea4411bd5ad84e4b436e03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f091d2f4e2348848c4312cde543dd92" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:06.432463 31389 mvcc.cc:205] Tried to move back new op lower bound from 7298467948223336448 to 7298467948029231104. Current Snapshot: MvccSnapshot[applied={T|T < 7298467948223336448}]
I20260619 07:02:06.433975 31388 mvcc.cc:205] Tried to move back new op lower bound from 7298467948223336448 to 7298467948029231104. Current Snapshot: MvccSnapshot[applied={T|T < 7298467948223336448}]
I20260619 07:02:06.439802 30897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:06.439970 30897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:06.442379 30897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed846ab761984620996df083c946bdd6: Sending DeleteTablet for 3 replicas of tablet 7c78c611183943ec879a4aadcd51ac52
I20260619 07:02:06.443184 31156 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c78c611183943ec879a4aadcd51ac52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:06 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:48140
I20260619 07:02:06.443186 31022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c78c611183943ec879a4aadcd51ac52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:06 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:36724
I20260619 07:02:06.443776 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 30959
I20260619 07:02:06.443825 31396 tablet_replica.cc:330] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: stopping tablet replica
I20260619 07:02:06.443804 31289 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c78c611183943ec879a4aadcd51ac52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:06 UTC) from {username='kudu', principal='kudu/127.11.106.126@KRBTEST.COM'} at 127.0.0.1:34992
I20260619 07:02:06.445048 31398 tablet_replica.cc:330] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: stopping tablet replica
I20260619 07:02:06.445333 31398 raft_consensus.cc:2229] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:02:06.445670 31398 raft_consensus.cc:2258] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:06.445824 31396 raft_consensus.cc:2229] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:02:06.446135 31396 raft_consensus.cc:2258] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:06.447109 31398 ts_tablet_manager.cc:1905] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:06.447516 31396 ts_tablet_manager.cc:1905] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:06.449940 31398 ts_tablet_manager.cc:1918] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:06.450045 31398 log.cc:1196] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/7c78c611183943ec879a4aadcd51ac52
I20260619 07:02:06.450219 31396 ts_tablet_manager.cc:1918] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:06.450343 31396 log.cc:1196] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/7c78c611183943ec879a4aadcd51ac52
I20260619 07:02:06.450466 31398 ts_tablet_manager.cc:1939] T 7c78c611183943ec879a4aadcd51ac52 P a2af32d6018a48559428acc6675f1171: Deleting consensus metadata
I20260619 07:02:06.450671 31396 ts_tablet_manager.cc:1939] T 7c78c611183943ec879a4aadcd51ac52 P 4f091d2f4e2348848c4312cde543dd92: Deleting consensus metadata
I20260619 07:02:06.451718 30884 catalog_manager.cc:4841] TS a2af32d6018a48559428acc6675f1171 (127.11.106.67:46605): tablet 7c78c611183943ec879a4aadcd51ac52 (table test-table [id=4345f47d2a1b48ba991cd6862ddc9442]) successfully deleted
I20260619 07:02:06.451689 30883 catalog_manager.cc:4841] TS 4f091d2f4e2348848c4312cde543dd92 (127.11.106.66:37571): tablet 7c78c611183943ec879a4aadcd51ac52 (table test-table [id=4345f47d2a1b48ba991cd6862ddc9442]) successfully deleted
I20260619 07:02:06.452095 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 31093
W20260619 07:02:06.452075 30884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:34035 (error 108)
I20260619 07:02:06.457082 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.456694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:34035 (local address 127.0.0.1:36754)
0619 07:02:06.456899 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:06.456976 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260619 07:02:06.457301 30884 catalog_manager.cc:4584] TS fe0d41299fea4411bd5ad84e4b436e03 (127.11.106.65:34035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c78c611183943ec879a4aadcd51ac52: Network error: Client connection negotiation failed: client connection to 127.11.106.65:34035: connect: Connection refused (error 111)
I20260619 07:02:06.458949 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 31227
I20260619 07:02:06.465183 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 30865
2026-06-19T07:02:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2514 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T07:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:06Z Disabled control of system clock
I20260619 07:02:06.496367 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46747
--webserver_interface=127.11.106.126
--webserver_port=0
--builtin_ntp_servers=127.11.106.84:36293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:46747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 07:02:06.604903 31403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:06.605159 31403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:06.605193 31403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:06.609107 31403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:06.609160 31403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:06.609176 31403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:06.609198 31403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:06.609215 31403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:06.613936 31403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:36293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:46747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:46747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:06.614928 31403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:06.616219 31403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:06.624799 31411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:06.624816 31409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:06.624857 31403 server_base.cc:1034] running on GCE node
W20260619 07:02:06.625028 31408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:06.625573 31403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:06.626673 31403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:06.627871 31403 hybrid_clock.cc:648] HybridClock initialized: now 1781852526627845 us; error 36 us; skew 500 ppm
I20260619 07:02:06.629804 31403 webserver.cc:460] Webserver started at http://127.11.106.126:45681/ using document root <none> and password file <none>
I20260619 07:02:06.630384 31403 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:06.630486 31403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:06.630724 31403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:06.632826 31403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "11319fa7126b45ccae67d1c76e4e583f"
format_stamp: "Formatted at 2026-06-19 07:02:06 on dist-test-slave-10d0"
server_key: "72c95e9e7f319c69ff7f49a5a3be27e0"
server_key_iv: "5ddebfab553a9f0e4f1dd24d702f4670"
server_key_version: "encryptionkey@0"
I20260619 07:02:06.633394 31403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "11319fa7126b45ccae67d1c76e4e583f"
format_stamp: "Formatted at 2026-06-19 07:02:06 on dist-test-slave-10d0"
server_key: "72c95e9e7f319c69ff7f49a5a3be27e0"
server_key_iv: "5ddebfab553a9f0e4f1dd24d702f4670"
server_key_version: "encryptionkey@0"
I20260619 07:02:06.637013 31403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 07:02:06.639403 31417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:06.640636 31403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:06.640769 31403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "11319fa7126b45ccae67d1c76e4e583f"
format_stamp: "Formatted at 2026-06-19 07:02:06 on dist-test-slave-10d0"
server_key: "72c95e9e7f319c69ff7f49a5a3be27e0"
server_key_iv: "5ddebfab553a9f0e4f1dd24d702f4670"
server_key_version: "encryptionkey@0"
I20260619 07:02:06.640873 31403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:06.656469 31403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:06.657235 31403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:06.657433 31403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:06.664600 31469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:46747 every 8 connection(s)
I20260619 07:02:06.664610 31403 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:46747
I20260619 07:02:06.665612 31403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:06.668731 31470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:06.671792 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 31403
I20260619 07:02:06.671938 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:06.672248 11689 external_mini_cluster.cc:1428] Setting key 58e374b4551bb643d555638f89940dca
I20260619 07:02:06.675009 31470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f: Bootstrap starting.
I20260619 07:02:06.677265 31470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:06.677986 31470 log.cc:824] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:06.679981 31470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f: No bootstrap required, opened a new log
I20260619 07:02:06.681627 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.673738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.11.106.126:46747)
0619 07:02:06.674393 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.674407 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.674442 (+ 35us) server_negotiation.cc:407] Connection header received
0619 07:02:06.675059 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.675112 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.675538 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.675874 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:06.676319 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.677379 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.678325 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.678789 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.678966 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.679833 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:06.679838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:06.679851 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:06.680289 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:06.680346 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:06.680366 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:06.680474 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.680691 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.680939 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":130,"threads_started":1}
I20260619 07:02:06.683974 31470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11319fa7126b45ccae67d1c76e4e583f" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46747 } }
I20260619 07:02:06.684257 31470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:06.684348 31470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11319fa7126b45ccae67d1c76e4e583f, State: Initialized, Role: FOLLOWER
I20260619 07:02:06.684832 31470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11319fa7126b45ccae67d1c76e4e583f" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46747 } }
I20260619 07:02:06.684981 31470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:06.685060 31470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:06.685160 31470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:06.686064 31470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11319fa7126b45ccae67d1c76e4e583f" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46747 } }
I20260619 07:02:06.686408 31470 leader_election.cc:302] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11319fa7126b45ccae67d1c76e4e583f; no voters:
I20260619 07:02:06.686659 31470 leader_election.cc:288] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:06.686785 31475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:06.687024 31475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [term 1 LEADER]: Becoming Leader. State: Replica: 11319fa7126b45ccae67d1c76e4e583f, State: Running, Role: LEADER
I20260619 07:02:06.687327 31475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11319fa7126b45ccae67d1c76e4e583f" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46747 } }
I20260619 07:02:06.688033 31470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:06.689246 31477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11319fa7126b45ccae67d1c76e4e583f. Latest consensus state: current_term: 1 leader_uuid: "11319fa7126b45ccae67d1c76e4e583f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11319fa7126b45ccae67d1c76e4e583f" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46747 } } }
I20260619 07:02:06.689375 31477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:06.689590 31476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11319fa7126b45ccae67d1c76e4e583f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11319fa7126b45ccae67d1c76e4e583f" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 46747 } } }
I20260619 07:02:06.689805 31484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:06.689678 31476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:06.693092 31484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:06.698609 31484 catalog_manager.cc:1347] Generated new cluster ID: 5dadf24f7e274487ab89409f321d57ea
I20260619 07:02:06.698675 31484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:06.707000 31484 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:06.708119 31484 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:06.713995 31484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f: Generated new TSK 0
I20260619 07:02:06.714733 31484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:06.722546 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.65:0
--local_ip_for_outbound_sockets=127.11.106.65
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46747
--builtin_ntp_servers=127.11.106.84:36293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 07:02:06.832831 31494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:06.833091 31494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:06.833164 31494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:06.837179 31494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:06.837251 31494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:06.837337 31494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.65
I20260619 07:02:06.842031 31494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:36293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.106.65
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.65
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:06.843039 31494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:06.844470 31494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:06.854168 31499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:06.854173 31500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:06.854322 31494 server_base.cc:1034] running on GCE node
W20260619 07:02:06.854523 31502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:06.854894 31494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:06.855538 31494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:06.856729 31494 hybrid_clock.cc:648] HybridClock initialized: now 1781852526856718 us; error 30 us; skew 500 ppm
I20260619 07:02:06.858492 31494 webserver.cc:460] Webserver started at http://127.11.106.65:35105/ using document root <none> and password file <none>
I20260619 07:02:06.859059 31494 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:06.859150 31494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:06.859381 31494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:06.861456 31494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/data/instance:
uuid: "d2bf745f90d74e498d88a2bf58b7767a"
format_stamp: "Formatted at 2026-06-19 07:02:06 on dist-test-slave-10d0"
server_key: "12434b9a9da1b0e94e6f0acf7b8e68f8"
server_key_iv: "30b7cf0c170296c226e1afe515e18054"
server_key_version: "encryptionkey@0"
I20260619 07:02:06.862021 31494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance:
uuid: "d2bf745f90d74e498d88a2bf58b7767a"
format_stamp: "Formatted at 2026-06-19 07:02:06 on dist-test-slave-10d0"
server_key: "12434b9a9da1b0e94e6f0acf7b8e68f8"
server_key_iv: "30b7cf0c170296c226e1afe515e18054"
server_key_version: "encryptionkey@0"
I20260619 07:02:06.865500 31494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:02:06.867909 31508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:06.869146 31494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:06.869277 31494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/wal
uuid: "d2bf745f90d74e498d88a2bf58b7767a"
format_stamp: "Formatted at 2026-06-19 07:02:06 on dist-test-slave-10d0"
server_key: "12434b9a9da1b0e94e6f0acf7b8e68f8"
server_key_iv: "30b7cf0c170296c226e1afe515e18054"
server_key_version: "encryptionkey@0"
I20260619 07:02:06.869387 31494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:06.907263 31494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:06.907992 31494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:06.908200 31494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:06.908957 31494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:06.909979 31494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:06.910025 31494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:06.910063 31494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:06.910079 31494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:06.918780 31494 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.65:40577
I20260619 07:02:06.918804 31621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.65:40577 every 8 connection(s)
I20260619 07:02:06.919735 31494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/data/info.pb
I20260619 07:02:06.929004 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.921627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.65:35881 (local address 127.11.106.126:46747)
0619 07:02:06.921774 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:06.921778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:06.922842 (+ 1064us) server_negotiation.cc:407] Connection header received
0619 07:02:06.924245 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:06.924248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:06.924309 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:06.924418 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:06.925385 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.925927 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.927082 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.927309 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.927423 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.928463 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:06.928467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:06.928469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:06.928489 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:06.928506 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:06.928514 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:06.928635 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:06.928799 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 07:02:06.928860 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 07:02:06.929354 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 31494
I20260619 07:02:06.929505 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-0/wal/instance
I20260619 07:02:06.929808 11689 external_mini_cluster.cc:1428] Setting key 386961b0b78b9ac3644520e551a442d2
I20260619 07:02:06.929997 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:06.922038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46747 (local address 127.11.106.65:35881)
0619 07:02:06.922692 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:06.922731 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:06.924022 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:06.924654 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:06.924662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:06.924749 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:06.925230 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.925242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.926062 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.926071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.926950 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:06.926960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:06.927463 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:06.927468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:06.927634 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:06.928214 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 07:02:06.928215 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:06.928254 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 07:02:06.928314 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:06.928636 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:06.928648 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:06.928657 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 07:02:06.928796 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 07:02:06.929088 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
I20260619 07:02:06.931772 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.66:0
--local_ip_for_outbound_sockets=127.11.106.66
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46747
--builtin_ntp_servers=127.11.106.84:36293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 07:02:06.931788 31622 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46747
I20260619 07:02:06.932036 31622 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:06.932587 31622 heartbeater.cc:511] Master 127.11.106.126:46747 requested a full tablet report, sending...
I20260619 07:02:06.934049 31434 ts_manager.cc:194] Registered new tserver with Master: d2bf745f90d74e498d88a2bf58b7767a (127.11.106.65:40577)
I20260619 07:02:06.935180 31434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.65:35881
W20260619 07:02:07.045324 31625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:07.045594 31625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:07.045663 31625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:07.050102 31625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:07.050176 31625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:07.050261 31625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.66
I20260619 07:02:07.055117 31625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:36293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.106.66
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.66
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:07.056190 31625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:07.057714 31625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:07.067376 31631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:07.067473 31630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:07.067490 31633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:07.067746 31625 server_base.cc:1034] running on GCE node
I20260619 07:02:07.068125 31625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:07.068846 31625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:07.070035 31625 hybrid_clock.cc:648] HybridClock initialized: now 1781852527070019 us; error 36 us; skew 500 ppm
I20260619 07:02:07.071902 31625 webserver.cc:460] Webserver started at http://127.11.106.66:37125/ using document root <none> and password file <none>
I20260619 07:02:07.072492 31625 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:07.072618 31625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:07.072858 31625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:07.074952 31625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/data/instance:
uuid: "2c95f634e6a947a89cd7c5f835c1e90a"
format_stamp: "Formatted at 2026-06-19 07:02:07 on dist-test-slave-10d0"
server_key: "3457df90358523324909530e2fbaaa00"
server_key_iv: "b0e825225e9f77c7e0502f4e7812934e"
server_key_version: "encryptionkey@0"
I20260619 07:02:07.075541 31625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance:
uuid: "2c95f634e6a947a89cd7c5f835c1e90a"
format_stamp: "Formatted at 2026-06-19 07:02:07 on dist-test-slave-10d0"
server_key: "3457df90358523324909530e2fbaaa00"
server_key_iv: "b0e825225e9f77c7e0502f4e7812934e"
server_key_version: "encryptionkey@0"
I20260619 07:02:07.079319 31625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 07:02:07.081915 31639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:07.083191 31625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:07.083340 31625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/wal
uuid: "2c95f634e6a947a89cd7c5f835c1e90a"
format_stamp: "Formatted at 2026-06-19 07:02:07 on dist-test-slave-10d0"
server_key: "3457df90358523324909530e2fbaaa00"
server_key_iv: "b0e825225e9f77c7e0502f4e7812934e"
server_key_version: "encryptionkey@0"
I20260619 07:02:07.083456 31625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:07.094328 31625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:07.094978 31625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:07.095165 31625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:07.095733 31625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:07.096711 31625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:07.096781 31625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:07.096844 31625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:07.096895 31625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:07.105763 31625 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.66:34353
I20260619 07:02:07.105798 31752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.66:34353 every 8 connection(s)
I20260619 07:02:07.106626 31625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/data/info.pb
I20260619 07:02:07.108256 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 31625
I20260619 07:02:07.108384 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/wal/instance
I20260619 07:02:07.108769 11689 external_mini_cluster.cc:1428] Setting key 1e7df5ba1faf0918632379240590802a
I20260619 07:02:07.111024 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.106.67:0
--local_ip_for_outbound_sockets=127.11.106.67
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46747
--builtin_ntp_servers=127.11.106.84:36293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 07:02:07.116430 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.108450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:46859 (local address 127.11.106.126:46747)
0619 07:02:07.108648 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:07.108654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:07.109911 (+ 1257us) server_negotiation.cc:407] Connection header received
0619 07:02:07.111573 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:07.111576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:07.111628 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:07.111736 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:07.112708 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.113233 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.114443 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.114659 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.114777 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.115925 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:07.115930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:07.115933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:07.115949 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:07.115968 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:07.115977 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:07.116059 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:07.116221 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 07:02:07.116281 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 07:02:07.117283 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.108905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46747 (local address 127.11.106.66:46859)
0619 07:02:07.109762 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:07.109818 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:07.111365 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:07.111937 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:07.111944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:07.112027 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:07.112503 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.112516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.113370 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.113374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.114300 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.114311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.114820 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.114824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.115013 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.115667 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 07:02:07.115669 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:07.115708 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 07:02:07.115769 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:07.116074 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:07.116083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:07.116089 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 07:02:07.116220 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 07:02:07.116488 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":138,"threads_started":1}
I20260619 07:02:07.118494 31753 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46747
I20260619 07:02:07.118728 31753 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:07.119263 31753 heartbeater.cc:511] Master 127.11.106.126:46747 requested a full tablet report, sending...
I20260619 07:02:07.120324 31434 ts_manager.cc:194] Registered new tserver with Master: 2c95f634e6a947a89cd7c5f835c1e90a (127.11.106.66:34353)
I20260619 07:02:07.120996 31434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.66:46859
W20260619 07:02:07.231223 31756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:07.231570 31756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:07.231650 31756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:07.237682 31756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:07.237761 31756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:07.237869 31756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.106.67
I20260619 07:02:07.243703 31756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:36293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.106.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.106.67
--webserver_port=0
--tserver_master_addrs=127.11.106.126:46747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.106.67
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:07.244940 31756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:07.246325 31756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:07.256074 31761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:07.256073 31762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:07.256083 31764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:07.256353 31756 server_base.cc:1034] running on GCE node
I20260619 07:02:07.256878 31756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:07.257496 31756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:07.258656 31756 hybrid_clock.cc:648] HybridClock initialized: now 1781852527258642 us; error 24 us; skew 500 ppm
I20260619 07:02:07.260330 31756 webserver.cc:460] Webserver started at http://127.11.106.67:35219/ using document root <none> and password file <none>
I20260619 07:02:07.261004 31756 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:07.261070 31756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:07.261324 31756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:07.263396 31756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/data/instance:
uuid: "2b6ed52926d44db98b26ef1953289c40"
format_stamp: "Formatted at 2026-06-19 07:02:07 on dist-test-slave-10d0"
server_key: "5bee9f69d6dee0f9770fccdc577cb45a"
server_key_iv: "7b80c0930c717e484ebdccc5f645164a"
server_key_version: "encryptionkey@0"
I20260619 07:02:07.263967 31756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance:
uuid: "2b6ed52926d44db98b26ef1953289c40"
format_stamp: "Formatted at 2026-06-19 07:02:07 on dist-test-slave-10d0"
server_key: "5bee9f69d6dee0f9770fccdc577cb45a"
server_key_iv: "7b80c0930c717e484ebdccc5f645164a"
server_key_version: "encryptionkey@0"
I20260619 07:02:07.267683 31756 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 07:02:07.270128 31770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:07.271366 31756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:07.271576 31756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/wal
uuid: "2b6ed52926d44db98b26ef1953289c40"
format_stamp: "Formatted at 2026-06-19 07:02:07 on dist-test-slave-10d0"
server_key: "5bee9f69d6dee0f9770fccdc577cb45a"
server_key_iv: "7b80c0930c717e484ebdccc5f645164a"
server_key_version: "encryptionkey@0"
I20260619 07:02:07.271692 31756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:07.289551 31756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:07.290269 31756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:07.290499 31756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:07.291132 31756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 07:02:07.292132 31756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 07:02:07.292182 31756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:07.292256 31756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 07:02:07.292310 31756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:07.301993 31756 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.67:41303
I20260619 07:02:07.302019 31883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.67:41303 every 8 connection(s)
I20260619 07:02:07.303033 31756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/data/info.pb
I20260619 07:02:07.307478 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 31756
I20260619 07:02:07.307576 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/wal/instance
I20260619 07:02:07.307900 11689 external_mini_cluster.cc:1428] Setting key 71c4b543fcf4cad35d25e6f67d569e70
I20260619 07:02:07.312839 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.305009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.67:33063 (local address 127.11.106.126:46747)
0619 07:02:07.305163 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:07.305167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:07.306270 (+ 1103us) server_negotiation.cc:407] Connection header received
0619 07:02:07.307746 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:07.307750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:07.307814 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:07.307920 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:07.309121 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.309771 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.311015 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.311236 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.311318 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.312386 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:07.312389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:07.312391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:07.312402 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:07.312416 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:07.312423 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:07.312477 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:07.312680 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 07:02:07.312730 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 07:02:07.313720 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.305451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:46747 (local address 127.11.106.67:33063)
0619 07:02:07.306139 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:07.306180 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:07.307442 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:07.308128 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:07.308136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:07.308257 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:07.308969 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.308986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.309897 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.309901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.310902 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.310912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.311367 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.311370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.311505 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.312150 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 07:02:07.312151 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:07.312193 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 07:02:07.312259 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:07.312494 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:07.312501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:07.312505 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 07:02:07.312663 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 07:02:07.312886 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
I20260619 07:02:07.314831 31884 heartbeater.cc:347] Connected to a master server at 127.11.106.126:46747
I20260619 07:02:07.315070 31884 heartbeater.cc:464] Registering TS with master...
I20260619 07:02:07.315588 31884 heartbeater.cc:511] Master 127.11.106.126:46747 requested a full tablet report, sending...
I20260619 07:02:07.316656 31434 ts_manager.cc:194] Registered new tserver with Master: 2b6ed52926d44db98b26ef1953289c40 (127.11.106.67:41303)
I20260619 07:02:07.317214 31434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.106.67:33063
I20260619 07:02:07.321791 11689 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 07:02:07.327212 31473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.323678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.11.106.126:46747)
0619 07:02:07.323796 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:07.323807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:07.323945 (+ 138us) server_negotiation.cc:407] Connection header received
0619 07:02:07.324049 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:07.324052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:07.324091 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:07.324172 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:07.324482 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.325040 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.325872 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.326073 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.326157 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.326860 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:07.326863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:07.326865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:07.326876 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:07.326893 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:07.326901 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:07.326958 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:07.327052 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 07:02:07.327098 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 07:02:07.330020 31434 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 07:02:07.333065 31434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 07:02:07.347697 31898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.341873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:41303 (local address 127.0.0.1:35596)
0619 07:02:07.342398 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:07.342448 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:07.342583 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:07.343618 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:07.343621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:07.343646 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:07.343813 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.343821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.345143 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.345148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.346322 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.346332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.347265 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.347267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.347346 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.347378 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 07:02:07.347456 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 07:02:07.347542 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":82,"spinlock_wait_cycles":11136,"thread_start_us":107,"threads_started":1}
I20260619 07:02:07.348093 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.341922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:40577 (local address 127.0.0.1:58492)
0619 07:02:07.342813 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:07.342830 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:07.342909 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:07.343620 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:07.343623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:07.343649 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:07.343838 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.343845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.345215 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.345218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.346711 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.346722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.347763 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.347766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.347856 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.347892 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 07:02:07.347946 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 07:02:07.347997 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":203,"spinlock_wait_cycles":3584,"thread_start_us":77,"threads_started":1}
I20260619 07:02:07.348229 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.342735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.11.106.65:40577)
0619 07:02:07.343120 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:07.343129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:07.343148 (+ 19us) server_negotiation.cc:407] Connection header received
0619 07:02:07.343211 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:07.343215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:07.343358 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:07.343523 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:07.343945 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.344879 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.346846 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.347614 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.347725 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.347912 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:07.347973 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 07:02:07.348073 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":93,"threads_started":1}
I20260619 07:02:07.348860 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.342362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.11.106.67:41303)
0619 07:02:07.342967 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:07.342980 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:07.343006 (+ 26us) server_negotiation.cc:407] Connection header received
0619 07:02:07.343085 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:07.343090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:07.343298 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:07.343457 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:07.343948 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.345039 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.346454 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.347147 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.347257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.347461 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:07.347514 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 07:02:07.347616 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":105,"threads_started":1}
I20260619 07:02:07.349924 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.341561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.66:34353 (local address 127.0.0.1:56446)
0619 07:02:07.342390 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:07.342442 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:07.342582 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:07.343470 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:07.343478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:07.343502 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:07.343687 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.343695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.345357 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.345360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.346713 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.346722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.349452 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.349456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.349569 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.349594 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 07:02:07.349648 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 07:02:07.349738 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":94,"spinlock_wait_cycles":9856,"thread_start_us":90,"threads_started":1}
I20260619 07:02:07.350594 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.341775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.11.106.66:34353)
0619 07:02:07.342967 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:07.342980 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:07.343006 (+ 26us) server_negotiation.cc:407] Connection header received
0619 07:02:07.343085 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:07.343090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:07.343298 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:07.343457 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:07.343866 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.345240 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.348400 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.349287 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.350211 (+ 924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.350348 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:07.350388 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 07:02:07.350458 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":138,"threads_started":1}
I20260619 07:02:07.351485 31556 tablet_service.cc:1467] Processing CreateTablet for tablet a884b477317d42f0aed3ea727dbe2db2 (DEFAULT_TABLE table=test-table [id=d80456a3abf14a98bb0433dc61a9469a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:07.351498 31818 tablet_service.cc:1467] Processing CreateTablet for tablet a884b477317d42f0aed3ea727dbe2db2 (DEFAULT_TABLE table=test-table [id=d80456a3abf14a98bb0433dc61a9469a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:07.352700 31556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a884b477317d42f0aed3ea727dbe2db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:07.352700 31818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a884b477317d42f0aed3ea727dbe2db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:07.354470 31687 tablet_service.cc:1467] Processing CreateTablet for tablet a884b477317d42f0aed3ea727dbe2db2 (DEFAULT_TABLE table=test-table [id=d80456a3abf14a98bb0433dc61a9469a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 07:02:07.355300 31687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a884b477317d42f0aed3ea727dbe2db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:07.361194 31901 tablet_bootstrap.cc:492] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a: Bootstrap starting.
I20260619 07:02:07.362418 31903 tablet_bootstrap.cc:492] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: Bootstrap starting.
I20260619 07:02:07.363667 31901 tablet_bootstrap.cc:654] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:07.364468 31903 tablet_bootstrap.cc:654] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:07.364568 31901 log.cc:824] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:07.360502 31902 tablet_bootstrap.cc:492] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: Bootstrap starting.
I20260619 07:02:07.365276 31903 log.cc:824] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:07.366675 31902 tablet_bootstrap.cc:654] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:07.366868 31901 tablet_bootstrap.cc:492] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a: No bootstrap required, opened a new log
I20260619 07:02:07.367107 31901 ts_tablet_manager.cc:1397] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 07:02:07.367272 31903 tablet_bootstrap.cc:492] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: No bootstrap required, opened a new log
I20260619 07:02:07.367479 31903 ts_tablet_manager.cc:1397] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 07:02:07.367659 31902 log.cc:824] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:07.369285 31902 tablet_bootstrap.cc:492] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: No bootstrap required, opened a new log
I20260619 07:02:07.369474 31902 ts_tablet_manager.cc:1397] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 07:02:07.371621 31903 raft_consensus.cc:348] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } }
I20260619 07:02:07.371621 31901 raft_consensus.cc:348] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } }
I20260619 07:02:07.371905 31901 raft_consensus.cc:374] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:07.371896 31903 raft_consensus.cc:374] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:07.371951 31903 raft_consensus.cc:729] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c95f634e6a947a89cd7c5f835c1e90a, State: Initialized, Role: FOLLOWER
I20260619 07:02:07.371951 31901 raft_consensus.cc:729] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bf745f90d74e498d88a2bf58b7767a, State: Initialized, Role: FOLLOWER
I20260619 07:02:07.372427 31903 consensus_queue.cc:260] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } }
I20260619 07:02:07.372427 31901 consensus_queue.cc:260] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } }
I20260619 07:02:07.372735 31902 raft_consensus.cc:348] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } }
I20260619 07:02:07.373055 31902 raft_consensus.cc:374] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:07.373126 31902 raft_consensus.cc:729] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b6ed52926d44db98b26ef1953289c40, State: Initialized, Role: FOLLOWER
I20260619 07:02:07.373250 31622 heartbeater.cc:503] Master 127.11.106.126:46747 was elected leader, sending a full tablet report...
I20260619 07:02:07.373440 31901 ts_tablet_manager.cc:1428] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260619 07:02:07.373440 31903 ts_tablet_manager.cc:1428] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 07:02:07.373584 31902 consensus_queue.cc:260] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } }
I20260619 07:02:07.373749 31753 heartbeater.cc:503] Master 127.11.106.126:46747 was elected leader, sending a full tablet report...
I20260619 07:02:07.374598 31902 ts_tablet_manager.cc:1428] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 07:02:07.374903 31884 heartbeater.cc:503] Master 127.11.106.126:46747 was elected leader, sending a full tablet report...
I20260619 07:02:07.375998 31908 raft_consensus.cc:482] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 07:02:07.376155 31908 raft_consensus.cc:504] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } }
I20260619 07:02:07.378218 31908 leader_election.cc:288] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b6ed52926d44db98b26ef1953289c40 (127.11.106.67:41303), d2bf745f90d74e498d88a2bf58b7767a (127.11.106.65:40577)
I20260619 07:02:07.382128 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.378676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:57777 (local address 127.11.106.67:41303)
0619 07:02:07.378902 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:07.378907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:07.378919 (+ 12us) server_negotiation.cc:407] Connection header received
0619 07:02:07.378966 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:07.378971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:07.379014 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:07.379096 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:07.379364 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.379997 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.381212 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.381600 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.381727 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.381775 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:07.381953 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 07:02:07.382017 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260619 07:02:07.382153 31755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.378407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.67:41303 (local address 127.11.106.66:57777)
0619 07:02:07.378639 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:07.378657 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:07.378739 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:07.379112 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:07.379115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:07.379129 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:07.379247 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.379255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.380114 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.380118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.381096 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.381106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.381722 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.381726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.381821 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.381843 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 07:02:07.381939 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 07:02:07.382010 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260619 07:02:07.382292 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.378881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.106.66:54597 (local address 127.11.106.65:40577)
0619 07:02:07.379001 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:07.379005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:07.379019 (+ 14us) server_negotiation.cc:407] Connection header received
0619 07:02:07.379060 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:07.379063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:07.379116 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:07.379203 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 07:02:07.379576 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.380147 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.381315 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.381717 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.381842 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.381886 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:07.382066 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 07:02:07.382131 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 07:02:07.382561 31910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.378565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:40577 (local address 127.11.106.66:54597)
0619 07:02:07.378815 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:07.378831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:07.378922 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:07.379318 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:07.379321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:07.379343 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 07:02:07.379469 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.379478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.380265 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.380268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.381195 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:07.381205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.381837 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.381841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:07.381935 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.381958 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 07:02:07.382050 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 07:02:07.382123 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
I20260619 07:02:07.382804 31838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884b477317d42f0aed3ea727dbe2db2" candidate_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6ed52926d44db98b26ef1953289c40" is_pre_election: true
I20260619 07:02:07.383082 31838 raft_consensus.cc:2454] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c95f634e6a947a89cd7c5f835c1e90a in term 0.
I20260619 07:02:07.383281 31576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884b477317d42f0aed3ea727dbe2db2" candidate_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bf745f90d74e498d88a2bf58b7767a" is_pre_election: true
I20260619 07:02:07.383526 31576 raft_consensus.cc:2454] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c95f634e6a947a89cd7c5f835c1e90a in term 0.
I20260619 07:02:07.383544 31642 leader_election.cc:302] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6ed52926d44db98b26ef1953289c40, 2c95f634e6a947a89cd7c5f835c1e90a; no voters:
I20260619 07:02:07.384083 31908 raft_consensus.cc:2781] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 07:02:07.384155 31908 raft_consensus.cc:482] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 07:02:07.384217 31908 raft_consensus.cc:3037] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:07.385316 31908 raft_consensus.cc:504] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } }
I20260619 07:02:07.385635 31908 leader_election.cc:288] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [CANDIDATE]: Term 1 election: Requested vote from peers 2b6ed52926d44db98b26ef1953289c40 (127.11.106.67:41303), d2bf745f90d74e498d88a2bf58b7767a (127.11.106.65:40577)
I20260619 07:02:07.385962 31838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884b477317d42f0aed3ea727dbe2db2" candidate_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b6ed52926d44db98b26ef1953289c40"
I20260619 07:02:07.385964 31576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a884b477317d42f0aed3ea727dbe2db2" candidate_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bf745f90d74e498d88a2bf58b7767a"
I20260619 07:02:07.386080 31838 raft_consensus.cc:3037] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:07.386080 31576 raft_consensus.cc:3037] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:07.387312 31576 raft_consensus.cc:2454] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c95f634e6a947a89cd7c5f835c1e90a in term 1.
I20260619 07:02:07.387312 31838 raft_consensus.cc:2454] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c95f634e6a947a89cd7c5f835c1e90a in term 1.
I20260619 07:02:07.387633 31642 leader_election.cc:302] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b6ed52926d44db98b26ef1953289c40, 2c95f634e6a947a89cd7c5f835c1e90a; no voters:
I20260619 07:02:07.387859 31908 raft_consensus.cc:2781] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:07.388034 31908 raft_consensus.cc:686] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 1 LEADER]: Becoming Leader. State: Replica: 2c95f634e6a947a89cd7c5f835c1e90a, State: Running, Role: LEADER
I20260619 07:02:07.388321 31908 consensus_queue.cc:237] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } }
I20260619 07:02:07.391548 31434 catalog_manager.cc:5495] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c95f634e6a947a89cd7c5f835c1e90a (127.11.106.66). New cstate: current_term: 1 leader_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c95f634e6a947a89cd7c5f835c1e90a" member_type: VOTER last_known_addr { host: "127.11.106.66" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 } health_report { overall_health: UNKNOWN } } }
I20260619 07:02:07.413813 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.410473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.11.106.66:34353)
0619 07:02:07.410607 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:07.410611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:07.410623 (+ 12us) server_negotiation.cc:407] Connection header received
0619 07:02:07.410697 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:07.410699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:07.410754 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:07.410854 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:07.411066 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.411595 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.412392 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:07.412701 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:07.412782 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:07.413042 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:07.413451 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:07.413575 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:07.413637 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 07:02:07.413684 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 07:02:07.419510 31838 raft_consensus.cc:1261] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [term 1 FOLLOWER]: Refusing update from remote peer 2c95f634e6a947a89cd7c5f835c1e90a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:07.419517 31576 raft_consensus.cc:1261] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a [term 1 FOLLOWER]: Refusing update from remote peer 2c95f634e6a947a89cd7c5f835c1e90a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:07.420207 31908 consensus_queue.cc:1035] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b6ed52926d44db98b26ef1953289c40" member_type: VOTER last_known_addr { host: "127.11.106.67" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:07.420409 31911 consensus_queue.cc:1035] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bf745f90d74e498d88a2bf58b7767a" member_type: VOTER last_known_addr { host: "127.11.106.65" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 07:02:07.420948 31623 tablet.cc:2369] T a884b477317d42f0aed3ea727dbe2db2 P d2bf745f90d74e498d88a2bf58b7767a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 07:02:07.427356 31916 mvcc.cc:205] Tried to move back new op lower bound from 7298467952303222784 to 7298467952185622528. Current Snapshot: MvccSnapshot[applied={T|T < 7298467952303222784}]
I20260619 07:02:07.428578 31914 mvcc.cc:205] Tried to move back new op lower bound from 7298467952303222784 to 7298467952185622528. Current Snapshot: MvccSnapshot[applied={T|T < 7298467952303222784}]
I20260619 07:02:07.428941 31915 mvcc.cc:205] Tried to move back new op lower bound from 7298467952303222784 to 7298467952185622528. Current Snapshot: MvccSnapshot[applied={T|T < 7298467952303222784}]
I20260619 07:02:07.437225 31433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:07.437409 31433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 07:02:07.439720 31433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11319fa7126b45ccae67d1c76e4e583f: Sending DeleteTablet for 3 replicas of tablet a884b477317d42f0aed3ea727dbe2db2
I20260619 07:02:07.440479 31687 tablet_service.cc:1514] Processing DeleteTablet for tablet a884b477317d42f0aed3ea727dbe2db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:07 UTC) from {username='slave'} at 127.0.0.1:56446
I20260619 07:02:07.440642 31556 tablet_service.cc:1514] Processing DeleteTablet for tablet a884b477317d42f0aed3ea727dbe2db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:07 UTC) from {username='slave'} at 127.0.0.1:58492
I20260619 07:02:07.441045 31818 tablet_service.cc:1514] Processing DeleteTablet for tablet a884b477317d42f0aed3ea727dbe2db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 07:02:07 UTC) from {username='slave'} at 127.0.0.1:35596
I20260619 07:02:07.441264 31924 tablet_replica.cc:330] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: stopping tablet replica
I20260619 07:02:07.441515 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 31494
I20260619 07:02:07.441517 31926 tablet_replica.cc:330] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: stopping tablet replica
I20260619 07:02:07.441723 31924 raft_consensus.cc:2229] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 1 LEADER]: Raft consensus shutting down.
I20260619 07:02:07.441794 31926 raft_consensus.cc:2229] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 07:02:07.442478 31926 raft_consensus.cc:2258] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:07.442513 31924 raft_consensus.cc:2258] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 07:02:07.444060 31926 ts_tablet_manager.cc:1905] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:07.444060 31924 ts_tablet_manager.cc:1905] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 07:02:07.446784 31926 ts_tablet_manager.cc:1918] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:07.446880 31926 log.cc:1196] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-2/wal/wals/a884b477317d42f0aed3ea727dbe2db2
I20260619 07:02:07.447190 31926 ts_tablet_manager.cc:1939] T a884b477317d42f0aed3ea727dbe2db2 P 2b6ed52926d44db98b26ef1953289c40: Deleting consensus metadata
I20260619 07:02:07.447961 31924 ts_tablet_manager.cc:1918] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 07:02:07.448117 31924 log.cc:1196] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: Deleting WAL directory at /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781852273483449-11689-0/minicluster-data/ts-1/wal/wals/a884b477317d42f0aed3ea727dbe2db2
I20260619 07:02:07.448385 31420 catalog_manager.cc:4841] TS 2b6ed52926d44db98b26ef1953289c40 (127.11.106.67:41303): tablet a884b477317d42f0aed3ea727dbe2db2 (table test-table [id=d80456a3abf14a98bb0433dc61a9469a]) successfully deleted
I20260619 07:02:07.448571 31924 ts_tablet_manager.cc:1939] T a884b477317d42f0aed3ea727dbe2db2 P 2c95f634e6a947a89cd7c5f835c1e90a: Deleting consensus metadata
I20260619 07:02:07.449810 31419 catalog_manager.cc:4841] TS 2c95f634e6a947a89cd7c5f835c1e90a (127.11.106.66:34353): tablet a884b477317d42f0aed3ea727dbe2db2 (table test-table [id=d80456a3abf14a98bb0433dc61a9469a]) successfully deleted
W20260619 07:02:07.451211 31421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.106.65:40577 (error 108)
I20260619 07:02:07.452039 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 31625
I20260619 07:02:07.452283 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:07.451964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.65:40577 (local address 127.0.0.1:58500)
0619 07:02:07.452096 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:07.452180 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.65:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 07:02:07.452874 31421 catalog_manager.cc:4584] TS d2bf745f90d74e498d88a2bf58b7767a (127.11.106.65:40577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a884b477317d42f0aed3ea727dbe2db2: Network error: Client connection negotiation failed: client connection to 127.11.106.65:40577: connect: Connection refused (error 111)
I20260619 07:02:07.457635 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 31756
I20260619 07:02:07.463131 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 31403
2026-06-19T07:02:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (995 ms)
[----------] 6 tests from EncryptionPolicyTest (10499 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:07 dist-test-slave-10d0 krb5kdc[31930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:07 dist-test-slave-10d0 krb5kdc[31930](info): set up 2 sockets
Jun 19 07:02:07 dist-test-slave-10d0 krb5kdc[31930](info): commencing operation
krb5kdc: starting...
W20260619 07:02:08.513401 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:08 dist-test-slave-10d0 krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:08.631808 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:08.741060 31943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:08.741364 31943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:08.741441 31943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:08.745431 31943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:08.745532 31943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:08.745573 31943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:08.745606 31943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:08.745636 31943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:08.750945 31943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:08.752043 31943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:08.753203 31943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:08.761435 31948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:08.761571 31943 server_base.cc:1034] running on GCE node
W20260619 07:02:08.761464 31951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:08.761438 31949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:08.762351 31943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:08.763681 31943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:08.764863 31943 hybrid_clock.cc:648] HybridClock initialized: now 1781852528764830 us; error 46 us; skew 500 ppm
Jun 19 07:02:08 dist-test-slave-10d0 krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:08.767968 31943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:08.768934 31943 webserver.cc:460] Webserver started at http://127.0.0.1:41113/ using document root <none> and password file <none>
I20260619 07:02:08.769512 31943 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:08.769629 31943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:08.769865 31943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:08.771947 31943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "d451fd6bb263476a92898fa02dc39e69"
format_stamp: "Formatted at 2026-06-19 07:02:08 on dist-test-slave-10d0"
server_key: "e93945e77c494649d13a909a48d73a42"
server_key_iv: "3f3aebe1a642127fa588a508d18e5775"
server_key_version: "encryptionkey@0"
I20260619 07:02:08.772526 31943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "d451fd6bb263476a92898fa02dc39e69"
format_stamp: "Formatted at 2026-06-19 07:02:08 on dist-test-slave-10d0"
server_key: "e93945e77c494649d13a909a48d73a42"
server_key_iv: "3f3aebe1a642127fa588a508d18e5775"
server_key_version: "encryptionkey@0"
I20260619 07:02:08.776265 31943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 07:02:08.778728 31958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:08.779927 31943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:08.780071 31943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "d451fd6bb263476a92898fa02dc39e69"
format_stamp: "Formatted at 2026-06-19 07:02:08 on dist-test-slave-10d0"
server_key: "e93945e77c494649d13a909a48d73a42"
server_key_iv: "3f3aebe1a642127fa588a508d18e5775"
server_key_version: "encryptionkey@0"
I20260619 07:02:08.780184 31943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:08.798938 31943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:08.802439 31943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:08.802654 31943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:08.809623 31943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39849
I20260619 07:02:08.809631 32010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39849 every 8 connection(s)
I20260619 07:02:08.810562 31943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:08.813293 32011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:08.817842 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 31943
I20260619 07:02:08.817979 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:08.818323 11689 external_mini_cluster.cc:1428] Setting key c3136fcd56636c63fb10bab062fd1068
I20260619 07:02:08.819252 32011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69: Bootstrap starting.
I20260619 07:02:08.821633 32011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:08.822382 32011 log.cc:824] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:08.824203 32011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69: No bootstrap required, opened a new log
Jun 19 07:02:08 dist-test-slave-10d0 krb5kdc[31930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 07:02:08.827771 32011 raft_consensus.cc:348] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d451fd6bb263476a92898fa02dc39e69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260619 07:02:08.828070 32011 raft_consensus.cc:374] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:08.828138 32011 raft_consensus.cc:729] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d451fd6bb263476a92898fa02dc39e69, State: Initialized, Role: FOLLOWER
I20260619 07:02:08.828703 32011 consensus_queue.cc:260] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d451fd6bb263476a92898fa02dc39e69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260619 07:02:08.828884 32011 raft_consensus.cc:388] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:08.828979 32011 raft_consensus.cc:482] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:08.829082 32011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:08.829950 32011 raft_consensus.cc:504] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d451fd6bb263476a92898fa02dc39e69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260619 07:02:08.830295 32011 leader_election.cc:302] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d451fd6bb263476a92898fa02dc39e69; no voters:
I20260619 07:02:08.830669 32011 leader_election.cc:288] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:08.830713 32016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:08.830892 32016 raft_consensus.cc:686] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [term 1 LEADER]: Becoming Leader. State: Replica: d451fd6bb263476a92898fa02dc39e69, State: Running, Role: LEADER
I20260619 07:02:08.831205 32016 consensus_queue.cc:237] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d451fd6bb263476a92898fa02dc39e69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } }
I20260619 07:02:08.831923 32011 sys_catalog.cc:564] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:08.832365 32018 sys_catalog.cc:455] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d451fd6bb263476a92898fa02dc39e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d451fd6bb263476a92898fa02dc39e69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } } }
I20260619 07:02:08.832484 32018 sys_catalog.cc:458] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:08.832932 32021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:08.833395 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:08.819773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45273 (local address 127.0.0.1:39849)
0619 07:02:08.820210 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:08.820227 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:08.820262 (+ 35us) server_negotiation.cc:407] Connection header received
0619 07:02:08.820947 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:08.820966 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:08.821310 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:08.821638 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:08.822440 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:08.823466 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:08.824330 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:08.824874 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:08.825042 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:08.827556 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:08.827562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:08.827575 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:08.827604 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:08.829590 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:08.830110 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:08.830122 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:08.830127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:08.830235 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:08.830624 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:08.830627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:08.830629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:08.831181 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:08.831428 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:08.832128 (+ 700us) server_negotiation.cc:299] Negotiation successful
0619 07:02:08.832383 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":129,"threads_started":1}
I20260619 07:02:08.834123 32017 sys_catalog.cc:455] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d451fd6bb263476a92898fa02dc39e69. Latest consensus state: current_term: 1 leader_uuid: "d451fd6bb263476a92898fa02dc39e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d451fd6bb263476a92898fa02dc39e69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39849 } } }
I20260619 07:02:08.834216 32017 sys_catalog.cc:458] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:08.836817 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:08.837136 32021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:08.843304 32021 catalog_manager.cc:1347] Generated new cluster ID: 7adf6b9ecb02442b84cf4211c35bbf5d
I20260619 07:02:08.843372 32021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:08.853422 32014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:08.838969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41361 (local address 127.0.0.1:39849)
0619 07:02:08.839072 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:08.839076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:08.839578 (+ 502us) server_negotiation.cc:407] Connection header received
0619 07:02:08.839690 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:08.839693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:08.839742 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:08.839824 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:08.840641 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:08.841501 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:08.842381 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:08.842625 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:08.842737 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:08.849650 (+ 6913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:08.849654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:08.849657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:08.849685 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:08.851354 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:08.851869 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:08.851872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:08.851874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:08.851961 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:08.852292 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:08.852295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:08.852297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:08.852510 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:08.852686 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:08.852992 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 07:02:08.853258 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 07:02:08.862555 32021 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:08.863669 32021 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:08.869359 32021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d451fd6bb263476a92898fa02dc39e69: Generated new TSK 0
I20260619 07:02:08.870057 32021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:08.874243 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 31943
W20260619 07:02:08.880075 31928 connection.cc:535] client connection to 127.0.0.1:39849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39849): Connection reset by peer (error 104)
2026-06-19T07:02:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:08 dist-test-slave-10d0 krb5kdc[32042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:08 dist-test-slave-10d0 krb5kdc[32042](info): set up 2 sockets
Jun 19 07:02:08 dist-test-slave-10d0 krb5kdc[32042](info): commencing operation
krb5kdc: starting...
W20260619 07:02:09.913259 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:09 dist-test-slave-10d0 krb5kdc[32042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:10.037410 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:10.149775 32055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:10.150100 32055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:10.150183 32055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:10.154294 32055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:10.154405 32055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:10.154449 32055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:10.154515 32055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:10.154556 32055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:10.160002 32055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:10.161126 32055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:10.162173 32055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:10.170818 32060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:10.170830 32061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:10.170811 32063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:10.170920 32055 server_base.cc:1034] running on GCE node
I20260619 07:02:10.171624 32055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:10.172808 32055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:10.173967 32055 hybrid_clock.cc:648] HybridClock initialized: now 1781852530173937 us; error 43 us; skew 500 ppm
Jun 19 07:02:10 dist-test-slave-10d0 krb5kdc[32042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:10.181335 32055 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:10.182291 32055 webserver.cc:460] Webserver started at http://127.0.0.1:33783/ using document root <none> and password file <none>
I20260619 07:02:10.182835 32055 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:10.182891 32055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:10.183070 32055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:10.185201 32055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e"
format_stamp: "Formatted at 2026-06-19 07:02:10 on dist-test-slave-10d0"
server_key: "8aadd18ca4fdc931b8177a6ef14c9f78"
server_key_iv: "23705f8316099f302ac395fc1d5e281a"
server_key_version: "encryptionkey@0"
I20260619 07:02:10.185730 32055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e"
format_stamp: "Formatted at 2026-06-19 07:02:10 on dist-test-slave-10d0"
server_key: "8aadd18ca4fdc931b8177a6ef14c9f78"
server_key_iv: "23705f8316099f302ac395fc1d5e281a"
server_key_version: "encryptionkey@0"
I20260619 07:02:10.189383 32055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:02:10.191751 32070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:10.192971 32055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:10.193070 32055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e"
format_stamp: "Formatted at 2026-06-19 07:02:10 on dist-test-slave-10d0"
server_key: "8aadd18ca4fdc931b8177a6ef14c9f78"
server_key_iv: "23705f8316099f302ac395fc1d5e281a"
server_key_version: "encryptionkey@0"
I20260619 07:02:10.193151 32055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:10.203585 32055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:10.206866 32055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:10.207059 32055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:10.216149 32055 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36229
I20260619 07:02:10.216799 32122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36229 every 8 connection(s)
I20260619 07:02:10.217391 32055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:10.220013 32123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:10.224038 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 32055
I20260619 07:02:10.224184 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:10.224447 11689 external_mini_cluster.cc:1428] Setting key a087fba68ed7e31b923d5044db66b552
I20260619 07:02:10.225894 32123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e: Bootstrap starting.
I20260619 07:02:10.228066 32123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:10.228858 32123 log.cc:824] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:10.230937 32123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e: No bootstrap required, opened a new log
I20260619 07:02:10.233747 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:10.225872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56089 (local address 127.0.0.1:36229)
0619 07:02:10.226300 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:10.226314 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:10.226350 (+ 36us) server_negotiation.cc:407] Connection header received
0619 07:02:10.226994 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:10.227011 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:10.227343 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:10.227693 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:10.228094 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:10.229296 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:10.230554 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:10.231004 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:10.231165 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:10.232038 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:10.232043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:10.232059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:10.232422 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:10.232473 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:10.232495 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:10.232672 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:10.232859 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 07:02:10.233024 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":133,"threads_started":1}
I20260619 07:02:10.234958 32123 raft_consensus.cc:348] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36229 } }
I20260619 07:02:10.235211 32123 raft_consensus.cc:374] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:10.235256 32123 raft_consensus.cc:729] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df68d5e8f1b84edfb7bbdcc9ab76596e, State: Initialized, Role: FOLLOWER
I20260619 07:02:10.235749 32123 consensus_queue.cc:260] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36229 } }
I20260619 07:02:10.235948 32123 raft_consensus.cc:388] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:10.236025 32123 raft_consensus.cc:482] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:10.236147 32123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:10.236766 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:10.237239 32123 raft_consensus.cc:504] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36229 } }
I20260619 07:02:10.237622 32123 leader_election.cc:302] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df68d5e8f1b84edfb7bbdcc9ab76596e; no voters:
I20260619 07:02:10.237955 32123 leader_election.cc:288] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:10.238068 32128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:10.238300 32128 raft_consensus.cc:686] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [term 1 LEADER]: Becoming Leader. State: Replica: df68d5e8f1b84edfb7bbdcc9ab76596e, State: Running, Role: LEADER
I20260619 07:02:10.238605 32128 consensus_queue.cc:237] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36229 } }
I20260619 07:02:10.239312 32123 sys_catalog.cc:564] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:10.239820 32134 sys_catalog.cc:455] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [sys.catalog]: SysCatalogTable state changed. Reason: New leader df68d5e8f1b84edfb7bbdcc9ab76596e. Latest consensus state: current_term: 1 leader_uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36229 } } }
I20260619 07:02:10.239943 32134 sys_catalog.cc:458] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:10.240226 32132 sys_catalog.cc:455] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df68d5e8f1b84edfb7bbdcc9ab76596e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36229 } } }
I20260619 07:02:10.240339 32132 sys_catalog.cc:458] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:10.242110 32139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:10.244668 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:10.239300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51627 (local address 127.0.0.1:36229)
0619 07:02:10.239412 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:10.239416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:10.240532 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 07:02:10.240721 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:10.240724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:10.240777 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:10.240864 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:10.241185 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:10.241891 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:10.242817 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:10.243149 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:10.243262 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:10.244170 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:10.244173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:10.244176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:10.244189 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:10.244216 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:10.244228 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:10.244320 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:10.244486 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 07:02:10.244520 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 07:02:10.245245 32139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:10.250782 32139 catalog_manager.cc:1347] Generated new cluster ID: 9959056bf7f24020b5a91dc97a7b3947
I20260619 07:02:10.250860 32139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:10.265630 32139 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:10.266498 32139 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:10.277582 32139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df68d5e8f1b84edfb7bbdcc9ab76596e: Generated new TSK 0
I20260619 07:02:10.278335 32139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:10.288272 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 32055
W20260619 07:02:10.294137 32040 connection.cc:535] client connection to 127.0.0.1:36229 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36229): Connection reset by peer (error 104)
2026-06-19T07:02:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:10 dist-test-slave-10d0 krb5kdc[32154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:10 dist-test-slave-10d0 krb5kdc[32154](info): set up 2 sockets
Jun 19 07:02:10 dist-test-slave-10d0 krb5kdc[32154](info): commencing operation
krb5kdc: starting...
W20260619 07:02:11.337420 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:11 dist-test-slave-10d0 krb5kdc[32154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:11.459280 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:11.573288 32167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:11.573539 32167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:11.573580 32167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:11.577482 32167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:11.577538 32167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:11.577558 32167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:11.577574 32167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:11.577590 32167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:11.582690 32167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:11.583675 32167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:11.584723 32167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:11.593089 32173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:11.593165 32175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:11.593228 32172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:11.593493 32167 server_base.cc:1034] running on GCE node
I20260619 07:02:11.594115 32167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:11.595201 32167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:11.596364 32167 hybrid_clock.cc:648] HybridClock initialized: now 1781852531596341 us; error 40 us; skew 500 ppm
Jun 19 07:02:11 dist-test-slave-10d0 krb5kdc[32154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:11.599309 32167 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:11.600224 32167 webserver.cc:460] Webserver started at http://127.0.0.1:43989/ using document root <none> and password file <none>
I20260619 07:02:11.600838 32167 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:11.600925 32167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:11.601163 32167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:11.603163 32167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "d039ea145b374a42a25f91670a28f114"
format_stamp: "Formatted at 2026-06-19 07:02:11 on dist-test-slave-10d0"
server_key: "865c17c4baad15ccb62ac1916b9c9a84"
server_key_iv: "e6b094360dd68760315b3db4172c3684"
server_key_version: "encryptionkey@0"
I20260619 07:02:11.603717 32167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "d039ea145b374a42a25f91670a28f114"
format_stamp: "Formatted at 2026-06-19 07:02:11 on dist-test-slave-10d0"
server_key: "865c17c4baad15ccb62ac1916b9c9a84"
server_key_iv: "e6b094360dd68760315b3db4172c3684"
server_key_version: "encryptionkey@0"
I20260619 07:02:11.607381 32167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 07:02:11.610065 32182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:11.611290 32167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 07:02:11.611429 32167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "d039ea145b374a42a25f91670a28f114"
format_stamp: "Formatted at 2026-06-19 07:02:11 on dist-test-slave-10d0"
server_key: "865c17c4baad15ccb62ac1916b9c9a84"
server_key_iv: "e6b094360dd68760315b3db4172c3684"
server_key_version: "encryptionkey@0"
I20260619 07:02:11.611536 32167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:11.616019 32167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:11.619225 32167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:11.619416 32167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:11.626649 32167 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40619
I20260619 07:02:11.626673 32234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40619 every 8 connection(s)
I20260619 07:02:11.627535 32167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:11.630244 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:11.635669 32235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114: Bootstrap starting.
I20260619 07:02:11.635917 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 32167
I20260619 07:02:11.636073 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:11.636369 11689 external_mini_cluster.cc:1428] Setting key ac763dee90873fe69c00ebbb41b6b0ae
I20260619 07:02:11.638324 32235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:11.639050 32235 log.cc:824] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:11.641304 32235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114: No bootstrap required, opened a new log
I20260619 07:02:11.641822 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:11.637729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35625 (local address 127.0.0.1:40619)
0619 07:02:11.638162 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:11.638177 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:11.638214 (+ 37us) server_negotiation.cc:407] Connection header received
0619 07:02:11.638908 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:11.638926 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:11.639256 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:11.639607 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:11.640290 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:11.640295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:11.640310 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:11.640320 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:11.640375 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:11.640399 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:11.640589 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:11.640772 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 07:02:11.640945 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260619 07:02:11.644585 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:11.645162 32235 raft_consensus.cc:348] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d039ea145b374a42a25f91670a28f114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260619 07:02:11.645365 32235 raft_consensus.cc:374] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:11.645423 32235 raft_consensus.cc:729] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d039ea145b374a42a25f91670a28f114, State: Initialized, Role: FOLLOWER
I20260619 07:02:11.645897 32235 consensus_queue.cc:260] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d039ea145b374a42a25f91670a28f114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260619 07:02:11.646052 32235 raft_consensus.cc:388] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:11.646118 32235 raft_consensus.cc:482] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:11.646212 32235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:11.647352 32235 raft_consensus.cc:504] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d039ea145b374a42a25f91670a28f114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260619 07:02:11.647746 32235 leader_election.cc:302] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d039ea145b374a42a25f91670a28f114; no voters:
I20260619 07:02:11.648077 32235 leader_election.cc:288] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:11.648175 32245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:11.648391 32245 raft_consensus.cc:686] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [term 1 LEADER]: Becoming Leader. State: Replica: d039ea145b374a42a25f91670a28f114, State: Running, Role: LEADER
I20260619 07:02:11.648527 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:11.646786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37241 (local address 127.0.0.1:40619)
0619 07:02:11.646904 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:11.646908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:11.647066 (+ 158us) server_negotiation.cc:407] Connection header received
0619 07:02:11.647173 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:11.647175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:11.647201 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:11.647265 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:11.647983 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:11.647985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:11.647987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:11.647998 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:11.648021 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:11.648031 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:11.648163 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:11.648356 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 07:02:11.648396 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 07:02:11.648794 32245 consensus_queue.cc:237] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d039ea145b374a42a25f91670a28f114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260619 07:02:11.649422 32235 sys_catalog.cc:564] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:11.650558 32246 sys_catalog.cc:455] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d039ea145b374a42a25f91670a28f114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d039ea145b374a42a25f91670a28f114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } } }
I20260619 07:02:11.650682 32246 sys_catalog.cc:458] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:11.650996 32247 sys_catalog.cc:455] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d039ea145b374a42a25f91670a28f114. Latest consensus state: current_term: 1 leader_uuid: "d039ea145b374a42a25f91670a28f114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d039ea145b374a42a25f91670a28f114" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } } }
I20260619 07:02:11.651093 32247 sys_catalog.cc:458] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114 [sys.catalog]: This master's current role is: LEADER
W20260619 07:02:11.654656 32260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:11.654762 32260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:11.654840 32254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:11.655504 32254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:11.661321 32254 catalog_manager.cc:1347] Generated new cluster ID: 98f630e7d6874ddcb87430b41d58734c
I20260619 07:02:11.661412 32254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:11.680740 32254 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:11.681074 32254 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:11.689837 32254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d039ea145b374a42a25f91670a28f114: Generated new TSK 0
I20260619 07:02:11.690559 32254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:11.698658 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 32167
2026-06-19T07:02:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:11 dist-test-slave-10d0 krb5kdc[32266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:11 dist-test-slave-10d0 krb5kdc[32266](info): set up 2 sockets
Jun 19 07:02:11 dist-test-slave-10d0 krb5kdc[32266](info): commencing operation
krb5kdc: starting...
W20260619 07:02:12.833199 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:12 dist-test-slave-10d0 krb5kdc[32266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:12.954981 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:13.063787 32279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:13.064095 32279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:13.064178 32279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:13.068341 32279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:13.068429 32279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:13.068467 32279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:13.068526 32279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:13.068586 32279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:13.073904 32279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:13.074981 32279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:13.076066 32279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:13.084702 32284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:13.084707 32287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:13.084708 32285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:13.084792 32279 server_base.cc:1034] running on GCE node
I20260619 07:02:13.085451 32279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:13.086536 32279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:13.087698 32279 hybrid_clock.cc:648] HybridClock initialized: now 1781852533087679 us; error 32 us; skew 500 ppm
Jun 19 07:02:13 dist-test-slave-10d0 krb5kdc[32266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:13.093433 32279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:13.094357 32279 webserver.cc:460] Webserver started at http://127.0.0.1:33755/ using document root <none> and password file <none>
I20260619 07:02:13.094908 32279 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:13.095000 32279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:13.095243 32279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:13.097376 32279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7"
format_stamp: "Formatted at 2026-06-19 07:02:13 on dist-test-slave-10d0"
server_key: "e9e50fc44b4188d364b6af939ef1a84d"
server_key_iv: "4f473f508af30859586d61cf99d554b6"
server_key_version: "encryptionkey@0"
I20260619 07:02:13.097934 32279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7"
format_stamp: "Formatted at 2026-06-19 07:02:13 on dist-test-slave-10d0"
server_key: "e9e50fc44b4188d364b6af939ef1a84d"
server_key_iv: "4f473f508af30859586d61cf99d554b6"
server_key_version: "encryptionkey@0"
I20260619 07:02:13.101692 32279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 07:02:13.104049 32294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:13.105582 32279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 07:02:13.105722 32279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7"
format_stamp: "Formatted at 2026-06-19 07:02:13 on dist-test-slave-10d0"
server_key: "e9e50fc44b4188d364b6af939ef1a84d"
server_key_iv: "4f473f508af30859586d61cf99d554b6"
server_key_version: "encryptionkey@0"
I20260619 07:02:13.105868 32279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:13.132139 32279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:13.135540 32279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:13.135751 32279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:13.143121 32279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35747
I20260619 07:02:13.143141 32346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35747 every 8 connection(s)
I20260619 07:02:13.144096 32279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:13.146797 32347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:13.151131 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 32279
I20260619 07:02:13.151289 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:13.151559 11689 external_mini_cluster.cc:1428] Setting key c3cf25ee616ba2f94e9c85b9b4db8267
I20260619 07:02:13.152697 32347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7: Bootstrap starting.
I20260619 07:02:13.155367 32347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:13.156219 32347 log.cc:824] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:13.158289 32347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7: No bootstrap required, opened a new log
Jun 19 07:02:13 dist-test-slave-10d0 krb5kdc[32266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 07:02:13.162593 32347 raft_consensus.cc:348] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260619 07:02:13.162849 32347 raft_consensus.cc:374] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:13.162909 32347 raft_consensus.cc:729] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41ebe84949b4ebe81cbaa1bebba8ae7, State: Initialized, Role: FOLLOWER
I20260619 07:02:13.163404 32347 consensus_queue.cc:260] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260619 07:02:13.163548 32347 raft_consensus.cc:388] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:13.163628 32347 raft_consensus.cc:482] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:13.163756 32347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:13.164892 32347 raft_consensus.cc:504] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260619 07:02:13.165241 32347 leader_election.cc:302] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a41ebe84949b4ebe81cbaa1bebba8ae7; no voters:
I20260619 07:02:13.165532 32347 leader_election.cc:288] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:13.165647 32352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:13.165866 32352 raft_consensus.cc:686] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [term 1 LEADER]: Becoming Leader. State: Replica: a41ebe84949b4ebe81cbaa1bebba8ae7, State: Running, Role: LEADER
I20260619 07:02:13.166182 32352 consensus_queue.cc:237] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } }
I20260619 07:02:13.166890 32347 sys_catalog.cc:564] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:13.167372 32353 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a41ebe84949b4ebe81cbaa1bebba8ae7. Latest consensus state: current_term: 1 leader_uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } } }
I20260619 07:02:13.167495 32353 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:13.167634 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:13.152999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54791 (local address 127.0.0.1:35747)
0619 07:02:13.153429 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:13.153458 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:13.153499 (+ 41us) server_negotiation.cc:407] Connection header received
0619 07:02:13.154331 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:13.154354 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:13.154787 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:13.155172 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:13.155936 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:13.157366 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:13.158247 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:13.158937 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:13.159098 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:13.161864 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:13.161870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:13.161883 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:13.161910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:13.164046 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:13.164624 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:13.164629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:13.164633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:13.164744 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:13.165084 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:13.165089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:13.165091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:13.165640 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:13.165860 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:13.166356 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 07:02:13.166652 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":111,"threads_started":1}
I20260619 07:02:13.167855 32354 sys_catalog.cc:455] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a41ebe84949b4ebe81cbaa1bebba8ae7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35747 } } }
I20260619 07:02:13.167943 32354 sys_catalog.cc:458] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:13.168254 32356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:13.170997 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:13.171430 32356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:13.177920 32356 catalog_manager.cc:1347] Generated new cluster ID: 4845e5b50f86458aa02624e42fc20b33
I20260619 07:02:13.177986 32356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:13.181936 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:13.173685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56973 (local address 127.0.0.1:35747)
0619 07:02:13.173802 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:13.173807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:13.174067 (+ 260us) server_negotiation.cc:407] Connection header received
0619 07:02:13.174193 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:13.174196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:13.174246 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:13.174336 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:13.174977 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:13.175485 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:13.176454 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:13.176794 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:13.176909 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:13.178501 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:13.178505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:13.178508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:13.178534 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:13.180255 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:13.180739 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:13.180743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:13.180745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:13.180833 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:13.181102 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:13.181105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:13.181107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:13.181313 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:13.181453 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:13.181666 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 07:02:13.181778 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 07:02:13.186769 32356 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:13.187613 32356 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:13.193203 32356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a41ebe84949b4ebe81cbaa1bebba8ae7: Generated new TSK 0
I20260619 07:02:13.194092 32356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:13.204819 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 32279
2026-06-19T07:02:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:13 dist-test-slave-10d0 krb5kdc[32378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:13 dist-test-slave-10d0 krb5kdc[32378](info): set up 2 sockets
Jun 19 07:02:13 dist-test-slave-10d0 krb5kdc[32378](info): commencing operation
krb5kdc: starting...
W20260619 07:02:14.269385 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:14 dist-test-slave-10d0 krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:14.391897 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:14.500379 32391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:14.500716 32391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:14.500765 32391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:14.504628 32391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:14.504675 32391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:14.504694 32391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:14.504710 32391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:14.504725 32391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:14.509785 32391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:14.510717 32391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:14.511646 32391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:14.520044 32396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:14.520046 32397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:14.520113 32391 server_base.cc:1034] running on GCE node
W20260619 07:02:14.520078 32399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:14.520824 32391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:14.521868 32391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:14.523046 32391 hybrid_clock.cc:648] HybridClock initialized: now 1781852534523012 us; error 47 us; skew 500 ppm
Jun 19 07:02:14 dist-test-slave-10d0 krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:14.525794 32391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:14.526678 32391 webserver.cc:460] Webserver started at http://127.0.0.1:40079/ using document root <none> and password file <none>
I20260619 07:02:14.527253 32391 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:14.527343 32391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:14.527576 32391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:14.529626 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6"
format_stamp: "Formatted at 2026-06-19 07:02:14 on dist-test-slave-10d0"
server_key: "e6bfded0ba6ccc400ee3ea84d5342ddf"
server_key_iv: "94f5a05b962a28561add4ce3a390fe90"
server_key_version: "encryptionkey@0"
I20260619 07:02:14.530184 32391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6"
format_stamp: "Formatted at 2026-06-19 07:02:14 on dist-test-slave-10d0"
server_key: "e6bfded0ba6ccc400ee3ea84d5342ddf"
server_key_iv: "94f5a05b962a28561add4ce3a390fe90"
server_key_version: "encryptionkey@0"
I20260619 07:02:14.533799 32391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:02:14.536151 32406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:14.537528 32391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:14.537775 32391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6"
format_stamp: "Formatted at 2026-06-19 07:02:14 on dist-test-slave-10d0"
server_key: "e6bfded0ba6ccc400ee3ea84d5342ddf"
server_key_iv: "94f5a05b962a28561add4ce3a390fe90"
server_key_version: "encryptionkey@0"
I20260619 07:02:14.537886 32391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:14.551823 32391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:14.554562 32391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:14.554749 32391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:14.561547 32391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45337
I20260619 07:02:14.561573 32458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45337 every 8 connection(s)
I20260619 07:02:14.562431 32391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:14.565105 32459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:14.567772 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 32391
I20260619 07:02:14.567922 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:14.568236 11689 external_mini_cluster.cc:1428] Setting key cc95f4fa9046e66a24c9c0aeff1e07f5
I20260619 07:02:14.571106 32459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6: Bootstrap starting.
I20260619 07:02:14.573252 32459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:14.573987 32459 log.cc:824] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:14.575874 32459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6: No bootstrap required, opened a new log
I20260619 07:02:14.577072 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:14.569660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54281 (local address 127.0.0.1:45337)
0619 07:02:14.570080 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:14.570094 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:14.570129 (+ 35us) server_negotiation.cc:407] Connection header received
0619 07:02:14.570717 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:14.570733 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:14.571076 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:14.571409 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:14.571807 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:14.572832 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:14.573784 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:14.574298 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:14.574516 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:14.575342 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:14.575347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:14.575359 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:14.575734 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:14.575784 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:14.575807 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:14.575918 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:14.576122 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 07:02:14.576341 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260619 07:02:14.579166 32459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260619 07:02:14.579417 32459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:14.579489 32459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef47a564a904c1b96b1c8fc66c3e5c6, State: Initialized, Role: FOLLOWER
I20260619 07:02:14.579977 32459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260619 07:02:14.580113 32459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:14.580199 32459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:14.580302 32459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:14.580472 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:14.581354 32459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260619 07:02:14.581710 32459 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef47a564a904c1b96b1c8fc66c3e5c6; no voters:
I20260619 07:02:14.582047 32459 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:14.582131 32468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:14.582356 32468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ef47a564a904c1b96b1c8fc66c3e5c6, State: Running, Role: LEADER
I20260619 07:02:14.582721 32468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } }
I20260619 07:02:14.583343 32459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:14.583962 32468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef47a564a904c1b96b1c8fc66c3e5c6. Latest consensus state: current_term: 1 leader_uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } } }
I20260619 07:02:14.584156 32468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:14.584398 32469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef47a564a904c1b96b1c8fc66c3e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45337 } } }
I20260619 07:02:14.584504 32469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:14.584640 32478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:14.587635 32462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:14.582886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51203 (local address 127.0.0.1:45337)
0619 07:02:14.582980 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:14.582984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:14.583001 (+ 17us) server_negotiation.cc:407] Connection header received
0619 07:02:14.583036 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:14.583038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:14.583078 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:14.583160 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:14.583545 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:14.584309 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:14.585602 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:14.585956 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:14.586058 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:14.587155 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:14.587159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:14.587162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:14.587174 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:14.587202 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:14.587216 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:14.587292 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:14.587445 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 07:02:14.587505 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 07:02:14.587867 32478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:14.593266 32478 catalog_manager.cc:1347] Generated new cluster ID: 9a0249067886463cb7cc9af0708af8a3
I20260619 07:02:14.593327 32478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:14.604111 32478 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:14.605108 32478 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:14.610946 32478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef47a564a904c1b96b1c8fc66c3e5c6: Generated new TSK 0
I20260619 07:02:14.611598 32478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:14.618878 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 32391
2026-06-19T07:02:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:14 dist-test-slave-10d0 krb5kdc[32490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:14 dist-test-slave-10d0 krb5kdc[32490](info): set up 2 sockets
Jun 19 07:02:14 dist-test-slave-10d0 krb5kdc[32490](info): commencing operation
krb5kdc: starting...
W20260619 07:02:15.661221 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:15 dist-test-slave-10d0 krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:15.779191 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:15.887914 32503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:15.888154 32503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:15.888195 32503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:15.892020 32503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:15.892072 32503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:15.892091 32503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:15.892107 32503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:15.892122 32503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:15.897217 32503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:15.898136 32503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:15.899068 32503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:15.907384 32511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:15.907456 32508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:15.907364 32509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:15.907615 32503 server_base.cc:1034] running on GCE node
I20260619 07:02:15.908205 32503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:15.909317 32503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:15.910501 32503 hybrid_clock.cc:648] HybridClock initialized: now 1781852535910476 us; error 47 us; skew 500 ppm
Jun 19 07:02:15 dist-test-slave-10d0 krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:15.913170 32503 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:15.914031 32503 webserver.cc:460] Webserver started at http://127.0.0.1:44953/ using document root <none> and password file <none>
I20260619 07:02:15.914574 32503 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:15.914635 32503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:15.914870 32503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:15.916893 32503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "7fa79b76fd764722a3c6a57586e74af4"
format_stamp: "Formatted at 2026-06-19 07:02:15 on dist-test-slave-10d0"
server_key: "b439f5018845560cbfa0c369f5ab2e91"
server_key_iv: "f4df31a6c7e8fc62ea15eb73a7fc494b"
server_key_version: "encryptionkey@0"
I20260619 07:02:15.917430 32503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "7fa79b76fd764722a3c6a57586e74af4"
format_stamp: "Formatted at 2026-06-19 07:02:15 on dist-test-slave-10d0"
server_key: "b439f5018845560cbfa0c369f5ab2e91"
server_key_iv: "f4df31a6c7e8fc62ea15eb73a7fc494b"
server_key_version: "encryptionkey@0"
I20260619 07:02:15.921058 32503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 07:02:15.923354 32518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:15.924479 32503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:15.924671 32503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "7fa79b76fd764722a3c6a57586e74af4"
format_stamp: "Formatted at 2026-06-19 07:02:15 on dist-test-slave-10d0"
server_key: "b439f5018845560cbfa0c369f5ab2e91"
server_key_iv: "f4df31a6c7e8fc62ea15eb73a7fc494b"
server_key_version: "encryptionkey@0"
I20260619 07:02:15.924780 32503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:15.929231 32503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:15.931917 32503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:15.932098 32503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:15.939247 32503 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40915
I20260619 07:02:15.939273 32570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40915 every 8 connection(s)
I20260619 07:02:15.940148 32503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:15.942761 32571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:15.944459 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 32503
I20260619 07:02:15.944625 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:15.944923 11689 external_mini_cluster.cc:1428] Setting key 9e13df2ba26f7c26958ae943df8104bb
I20260619 07:02:15.949034 32571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4: Bootstrap starting.
I20260619 07:02:15.949954 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:15.946281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39613 (local address 127.0.0.1:40915)
0619 07:02:15.946715 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:15.946734 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:15.946770 (+ 36us) server_negotiation.cc:407] Connection header received
0619 07:02:15.947383 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:15.947400 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:15.947676 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:15.948009 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:15.948828 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:15.948833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:15.948844 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:15.948854 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:15.948906 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:15.948925 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:15.949073 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:15.949143 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 07:02:15.949298 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":130,"threads_started":1}
I20260619 07:02:15.951442 32571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:15.952299 32571 log.cc:824] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:15.952800 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:15.954241 32571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4: No bootstrap required, opened a new log
I20260619 07:02:15.956368 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:15.954805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57555 (local address 127.0.0.1:40915)
0619 07:02:15.954926 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:15.954931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:15.955089 (+ 158us) server_negotiation.cc:407] Connection header received
0619 07:02:15.955219 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:15.955221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:15.955247 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:15.955308 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:15.955998 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:15.956001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:15.956003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:15.956013 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:15.956036 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:15.956047 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:15.956115 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:15.956216 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 07:02:15.956243 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 07:02:15.958379 32571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa79b76fd764722a3c6a57586e74af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260619 07:02:15.958611 32571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:15.958678 32571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa79b76fd764722a3c6a57586e74af4, State: Initialized, Role: FOLLOWER
I20260619 07:02:15.959077 32571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa79b76fd764722a3c6a57586e74af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260619 07:02:15.959226 32571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:15.959295 32571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:15.959393 32571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:15.960250 32571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa79b76fd764722a3c6a57586e74af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260619 07:02:15.960646 32571 leader_election.cc:302] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fa79b76fd764722a3c6a57586e74af4; no voters:
I20260619 07:02:15.960956 32571 leader_election.cc:288] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:15.961069 32581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:15.961329 32581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa79b76fd764722a3c6a57586e74af4, State: Running, Role: LEADER
I20260619 07:02:15.961658 32581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa79b76fd764722a3c6a57586e74af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260619 07:02:15.962218 32571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:15.963279 32582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fa79b76fd764722a3c6a57586e74af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa79b76fd764722a3c6a57586e74af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } } }
I20260619 07:02:15.963407 32582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:15.963725 32583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fa79b76fd764722a3c6a57586e74af4. Latest consensus state: current_term: 1 leader_uuid: "7fa79b76fd764722a3c6a57586e74af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa79b76fd764722a3c6a57586e74af4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } } }
I20260619 07:02:15.963892 32583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:15.963794 32590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:15.967247 32590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:15.972761 32590 catalog_manager.cc:1347] Generated new cluster ID: 1314464a6cdc46c592351e56a7e2f06a
I20260619 07:02:15.972824 32590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:15.997121 32590 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:15.997440 32590 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:16.006080 32590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fa79b76fd764722a3c6a57586e74af4: Generated new TSK 0
I20260619 07:02:16.006806 32590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:16.013821 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 32503
2026-06-19T07:02:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:16 dist-test-slave-10d0 krb5kdc[32602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:16 dist-test-slave-10d0 krb5kdc[32602](info): set up 2 sockets
Jun 19 07:02:16 dist-test-slave-10d0 krb5kdc[32602](info): commencing operation
krb5kdc: starting...
W20260619 07:02:17.069486 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:17 dist-test-slave-10d0 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:17.189815 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:17.299801 32615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:17.300066 32615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:17.300109 32615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:17.304116 32615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:17.304177 32615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:17.304196 32615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:17.304214 32615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:17.304256 32615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:17.309592 32615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:17.310678 32615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:17.311728 32615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:17.320524 32623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:17.320559 32621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:17.320619 32615 server_base.cc:1034] running on GCE node
W20260619 07:02:17.320605 32620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:17.321393 32615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:17.322537 32615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:17.323720 32615 hybrid_clock.cc:648] HybridClock initialized: now 1781852537323688 us; error 46 us; skew 500 ppm
Jun 19 07:02:17 dist-test-slave-10d0 krb5kdc[32602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:17.326462 32615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:17.327347 32615 webserver.cc:460] Webserver started at http://127.0.0.1:36239/ using document root <none> and password file <none>
I20260619 07:02:17.327891 32615 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:17.327993 32615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:17.328233 32615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:17.330262 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "bccd1fb8350e4ae78939ad0a768d8abc"
format_stamp: "Formatted at 2026-06-19 07:02:17 on dist-test-slave-10d0"
server_key: "1a472c15b27e9d8312163ef8c06eebfd"
server_key_iv: "b613ad5b0d4574ec5ec5071eb7f7fc76"
server_key_version: "encryptionkey@0"
I20260619 07:02:17.330804 32615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "bccd1fb8350e4ae78939ad0a768d8abc"
format_stamp: "Formatted at 2026-06-19 07:02:17 on dist-test-slave-10d0"
server_key: "1a472c15b27e9d8312163ef8c06eebfd"
server_key_iv: "b613ad5b0d4574ec5ec5071eb7f7fc76"
server_key_version: "encryptionkey@0"
I20260619 07:02:17.334404 32615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 07:02:17.336824 32630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:17.337984 32615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 07:02:17.338135 32615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "bccd1fb8350e4ae78939ad0a768d8abc"
format_stamp: "Formatted at 2026-06-19 07:02:17 on dist-test-slave-10d0"
server_key: "1a472c15b27e9d8312163ef8c06eebfd"
server_key_iv: "b613ad5b0d4574ec5ec5071eb7f7fc76"
server_key_version: "encryptionkey@0"
I20260619 07:02:17.338238 32615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:17.359390 32615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:17.362566 32615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:17.362807 32615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:17.370582 32615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32969
I20260619 07:02:17.370605 32682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32969 every 8 connection(s)
I20260619 07:02:17.371721 32615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:17.374637 32683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:17.376219 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 32615
I20260619 07:02:17.376368 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:17.376719 11689 external_mini_cluster.cc:1428] Setting key 306d063f9854b7a9383c14d2ea44c1d7
I20260619 07:02:17.381163 32683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc: Bootstrap starting.
I20260619 07:02:17.383705 32683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:17.384636 32683 log.cc:824] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc: Log is configured to *not* fsync() on all Append() calls
Jun 19 07:02:17 dist-test-slave-10d0 krb5kdc[32602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 07:02:17.386670 32683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc: No bootstrap required, opened a new log
I20260619 07:02:17.390062 32683 raft_consensus.cc:348] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccd1fb8350e4ae78939ad0a768d8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260619 07:02:17.390314 32683 raft_consensus.cc:374] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:17.390419 32683 raft_consensus.cc:729] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccd1fb8350e4ae78939ad0a768d8abc, State: Initialized, Role: FOLLOWER
I20260619 07:02:17.390951 32683 consensus_queue.cc:260] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccd1fb8350e4ae78939ad0a768d8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260619 07:02:17.391109 32683 raft_consensus.cc:388] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:17.391193 32683 raft_consensus.cc:482] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:17.391294 32683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:17.391661 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:17.378210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:35631 (local address 127.0.0.1:32969)
0619 07:02:17.378620 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:17.378638 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:17.378684 (+ 46us) server_negotiation.cc:407] Connection header received
0619 07:02:17.379327 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:17.379345 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:17.379698 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:17.379987 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:17.380961 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:17.382022 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:17.383224 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:17.383713 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:17.383895 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:17.386774 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:17.386782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:17.386799 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:17.386835 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:17.388599 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:17.389115 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:17.389120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:17.389125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:17.389233 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:17.389577 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:17.389585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:17.389586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:17.389966 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:17.390195 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:17.390526 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 07:02:17.390776 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":142,"threads_started":1}
I20260619 07:02:17.392303 32683 raft_consensus.cc:504] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccd1fb8350e4ae78939ad0a768d8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260619 07:02:17.392717 32683 leader_election.cc:302] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bccd1fb8350e4ae78939ad0a768d8abc; no voters:
I20260619 07:02:17.393080 32683 leader_election.cc:288] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:17.393164 32688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:17.393394 32688 raft_consensus.cc:686] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [term 1 LEADER]: Becoming Leader. State: Replica: bccd1fb8350e4ae78939ad0a768d8abc, State: Running, Role: LEADER
I20260619 07:02:17.394061 32688 consensus_queue.cc:237] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccd1fb8350e4ae78939ad0a768d8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260619 07:02:17.394378 32683 sys_catalog.cc:564] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:17.394853 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:17.395190 32688 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bccd1fb8350e4ae78939ad0a768d8abc. Latest consensus state: current_term: 1 leader_uuid: "bccd1fb8350e4ae78939ad0a768d8abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccd1fb8350e4ae78939ad0a768d8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260619 07:02:17.395300 32688 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:17.397856 32689 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bccd1fb8350e4ae78939ad0a768d8abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccd1fb8350e4ae78939ad0a768d8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260619 07:02:17.398034 32689 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc [sys.catalog]: This master's current role is: LEADER
W20260619 07:02:17.399529 32707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:17.399616 32707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:17.399730 32696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:17.400646 32696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:17.406458 32686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:17.397463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:49731 (local address 127.0.0.1:32969)
0619 07:02:17.397585 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:17.397589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:17.398037 (+ 448us) server_negotiation.cc:407] Connection header received
0619 07:02:17.398259 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:17.398262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:17.398313 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:17.398406 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:17.399174 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:17.399927 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:17.401080 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:17.401433 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:17.401553 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:17.403095 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:17.403099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:17.403102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:17.403127 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:17.404589 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:17.405055 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:17.405058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:17.405060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:17.405153 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:17.405595 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:17.405599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:17.405601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:17.405774 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:17.405921 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:17.406194 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 07:02:17.406287 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 07:02:17.407418 32696 catalog_manager.cc:1347] Generated new cluster ID: 5587de0baabe45539053e5c40bd8190f
I20260619 07:02:17.407503 32696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:17.440199 32696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:17.441260 32696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:17.451236 32696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bccd1fb8350e4ae78939ad0a768d8abc: Generated new TSK 0
I20260619 07:02:17.451990 32696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:17.460618 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 32615
W20260619 07:02:17.466279 32600 connection.cc:535] client connection to 127.0.0.1:32969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32969): Connection reset by peer (error 104)
2026-06-19T07:02:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:17 dist-test-slave-10d0 krb5kdc[32714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:17 dist-test-slave-10d0 krb5kdc[32714](info): set up 2 sockets
Jun 19 07:02:17 dist-test-slave-10d0 krb5kdc[32714](info): commencing operation
krb5kdc: starting...
W20260619 07:02:18.509235 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:18 dist-test-slave-10d0 krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:18.631582 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:18.743048 32727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:18.743309 32727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:18.743351 32727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:18.747427 32727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:18.747491 32727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:18.747511 32727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:18.747527 32727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:18.747542 32727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:18.752743 32727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:18.753819 32727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:18.754864 32727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:18.763373 32733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:18.763379 32735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:18.763586 32727 server_base.cc:1034] running on GCE node
W20260619 07:02:18.763836 32732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:18.764355 32727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:18.765480 32727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:18.766656 32727 hybrid_clock.cc:648] HybridClock initialized: now 1781852538766637 us; error 34 us; skew 500 ppm
Jun 19 07:02:18 dist-test-slave-10d0 krb5kdc[32714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:18.780185 32727 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:18.781191 32727 webserver.cc:460] Webserver started at http://127.0.0.1:33325/ using document root <none> and password file <none>
I20260619 07:02:18.781762 32727 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:18.781848 32727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:18.782087 32727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:18.784144 32727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "a2249e260058433a99ad1ad2cb3f9c85"
format_stamp: "Formatted at 2026-06-19 07:02:18 on dist-test-slave-10d0"
server_key: "5a7dd868756cdad93f9db085c5a2c635"
server_key_iv: "a3a47250055aa9f751e687c32cd2d59e"
server_key_version: "encryptionkey@0"
I20260619 07:02:18.784756 32727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "a2249e260058433a99ad1ad2cb3f9c85"
format_stamp: "Formatted at 2026-06-19 07:02:18 on dist-test-slave-10d0"
server_key: "5a7dd868756cdad93f9db085c5a2c635"
server_key_iv: "a3a47250055aa9f751e687c32cd2d59e"
server_key_version: "encryptionkey@0"
I20260619 07:02:18.788421 32727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 07:02:18.790805 32742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:18.791968 32727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:18.792114 32727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "a2249e260058433a99ad1ad2cb3f9c85"
format_stamp: "Formatted at 2026-06-19 07:02:18 on dist-test-slave-10d0"
server_key: "5a7dd868756cdad93f9db085c5a2c635"
server_key_iv: "a3a47250055aa9f751e687c32cd2d59e"
server_key_version: "encryptionkey@0"
I20260619 07:02:18.792215 32727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:18.817546 32727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:18.820890 32727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:18.821087 32727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:18.828107 32727 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38305
I20260619 07:02:18.828131 326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38305 every 8 connection(s)
I20260619 07:02:18.829104 32727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:18.831773 327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:18.837361 327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85: Bootstrap starting.
I20260619 07:02:18.837841 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 32727
I20260619 07:02:18.837996 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:18.838284 11689 external_mini_cluster.cc:1428] Setting key 7057f2425f46f0f315b79aafef88ec1f
I20260619 07:02:18.840091 327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:18.841006 327 log.cc:824] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:18.843153 327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85: No bootstrap required, opened a new log
I20260619 07:02:18.847352 327 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e260058433a99ad1ad2cb3f9c85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260619 07:02:18.847623 327 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:18.847692 327 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2249e260058433a99ad1ad2cb3f9c85, State: Initialized, Role: FOLLOWER
I20260619 07:02:18.848156 327 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e260058433a99ad1ad2cb3f9c85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260619 07:02:18.848280 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:18.839690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:42519 (local address 127.0.0.1:38305)
0619 07:02:18.840130 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:18.840151 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:18.840194 (+ 43us) server_negotiation.cc:407] Connection header received
0619 07:02:18.841061 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:18.841088 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:18.841526 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:18.841868 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:18.842272 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:18.843693 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:18.844587 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:18.845300 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:18.845463 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:18.846275 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:18.846281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:18.846297 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:18.846804 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:18.846873 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:18.846904 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:18.847040 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:18.847180 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 07:02:18.847307 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":102,"threads_started":1}
I20260619 07:02:18.848310 327 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:18.848369 327 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:18.848453 327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:18.849383 327 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e260058433a99ad1ad2cb3f9c85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260619 07:02:18.849742 327 leader_election.cc:302] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2249e260058433a99ad1ad2cb3f9c85; no voters:
I20260619 07:02:18.850068 327 leader_election.cc:288] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:18.850185 332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:18.850423 332 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [term 1 LEADER]: Becoming Leader. State: Replica: a2249e260058433a99ad1ad2cb3f9c85, State: Running, Role: LEADER
I20260619 07:02:18.850831 332 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e260058433a99ad1ad2cb3f9c85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260619 07:02:18.851359 327 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:18.851733 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:18.853214 333 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2249e260058433a99ad1ad2cb3f9c85. Latest consensus state: current_term: 1 leader_uuid: "a2249e260058433a99ad1ad2cb3f9c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e260058433a99ad1ad2cb3f9c85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260619 07:02:18.853359 333 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:18.853958 334 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2249e260058433a99ad1ad2cb3f9c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e260058433a99ad1ad2cb3f9c85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260619 07:02:18.854239 334 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85 [sys.catalog]: This master's current role is: LEADER
W20260619 07:02:18.856630 350 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:18.856714 350 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:18.856830 351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:18.857787 351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:18.858906 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:18.854058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:48243 (local address 127.0.0.1:38305)
0619 07:02:18.854184 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:18.854188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:18.854474 (+ 286us) server_negotiation.cc:407] Connection header received
0619 07:02:18.854593 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:18.854598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:18.854653 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:18.854730 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:18.855089 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:18.855782 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:18.856914 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:18.857274 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:18.857403 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:18.858369 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:18.858373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:18.858376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:18.858392 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:18.858421 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:18.858435 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:18.858538 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:18.858719 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 07:02:18.858756 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 07:02:18.863782 351 catalog_manager.cc:1347] Generated new cluster ID: f067bdb77c10445dbf468311e857c62b
I20260619 07:02:18.863874 351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:18.875036 351 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:18.876299 351 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:18.886219 351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2249e260058433a99ad1ad2cb3f9c85: Generated new TSK 0
I20260619 07:02:18.887051 351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:18.894950 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 32727
W20260619 07:02:18.900722 32712 connection.cc:535] client connection to 127.0.0.1:38305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38305): Connection reset by peer (error 104)
2026-06-19T07:02:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:18 dist-test-slave-10d0 krb5kdc[358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:18 dist-test-slave-10d0 krb5kdc[358](info): set up 2 sockets
Jun 19 07:02:18 dist-test-slave-10d0 krb5kdc[358](info): commencing operation
krb5kdc: starting...
W20260619 07:02:19.945393 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:20 dist-test-slave-10d0 krb5kdc[358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:20.067658 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:20.178188 371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:20.178476 371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:20.178519 371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:20.182554 371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:20.182612 371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:20.182632 371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:20.182648 371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:20.182675 371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:20.187851 371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:20.188874 371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:20.189895 371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:20.198935 379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:20.198940 377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:20.199103 371 server_base.cc:1034] running on GCE node
W20260619 07:02:20.198944 376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:20.199738 371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:20.200909 371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:20.202076 371 hybrid_clock.cc:648] HybridClock initialized: now 1781852540202046 us; error 48 us; skew 500 ppm
Jun 19 07:02:20 dist-test-slave-10d0 krb5kdc[358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:20.204856 371 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:20.205788 371 webserver.cc:460] Webserver started at http://127.0.0.1:33271/ using document root <none> and password file <none>
I20260619 07:02:20.206382 371 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:20.206455 371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:20.206682 371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:20.209169 371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "c63cd279a6bc4a36979d5fce9907aa50"
format_stamp: "Formatted at 2026-06-19 07:02:20 on dist-test-slave-10d0"
server_key: "7399e1015cdca1d18eaf60ad30a1311d"
server_key_iv: "629097fd439aed083b22d866dc09af48"
server_key_version: "encryptionkey@0"
I20260619 07:02:20.209946 371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "c63cd279a6bc4a36979d5fce9907aa50"
format_stamp: "Formatted at 2026-06-19 07:02:20 on dist-test-slave-10d0"
server_key: "7399e1015cdca1d18eaf60ad30a1311d"
server_key_iv: "629097fd439aed083b22d866dc09af48"
server_key_version: "encryptionkey@0"
I20260619 07:02:20.215457 371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 07:02:20.218250 386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:20.219458 371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:20.219602 371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "c63cd279a6bc4a36979d5fce9907aa50"
format_stamp: "Formatted at 2026-06-19 07:02:20 on dist-test-slave-10d0"
server_key: "7399e1015cdca1d18eaf60ad30a1311d"
server_key_iv: "629097fd439aed083b22d866dc09af48"
server_key_version: "encryptionkey@0"
I20260619 07:02:20.219714 371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:20.224414 371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:20.227634 371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:20.227777 371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:20.234771 371 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40497
I20260619 07:02:20.234781 440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40497 every 8 connection(s)
I20260619 07:02:20.235728 371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:20.238483 441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:20.243841 441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50: Bootstrap starting.
I20260619 07:02:20.244333 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 371
I20260619 07:02:20.244472 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:20.244783 11689 external_mini_cluster.cc:1428] Setting key 59b3cb2b76f68bfba4854a871a8b1b37
I20260619 07:02:20.246672 441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:20.247390 441 log.cc:824] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:20.249722 441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50: No bootstrap required, opened a new log
I20260619 07:02:20.250448 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:20.246164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:58685 (local address 127.0.0.1:40497)
0619 07:02:20.246628 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:20.246644 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:20.246681 (+ 37us) server_negotiation.cc:407] Connection header received
0619 07:02:20.247392 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:20.247411 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:20.247789 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:20.248199 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:20.249062 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:20.249067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:20.249080 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:20.249090 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:20.249142 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:20.249161 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:20.249254 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:20.249431 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 07:02:20.249641 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":125,"threads_started":1}
I20260619 07:02:20.253614 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:20.253738 441 raft_consensus.cc:348] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63cd279a6bc4a36979d5fce9907aa50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260619 07:02:20.253989 441 raft_consensus.cc:374] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:20.254031 441 raft_consensus.cc:729] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cd279a6bc4a36979d5fce9907aa50, State: Initialized, Role: FOLLOWER
I20260619 07:02:20.254527 441 consensus_queue.cc:260] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63cd279a6bc4a36979d5fce9907aa50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260619 07:02:20.254693 441 raft_consensus.cc:388] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:20.254760 441 raft_consensus.cc:482] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:20.254858 441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:20.256008 441 raft_consensus.cc:504] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63cd279a6bc4a36979d5fce9907aa50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260619 07:02:20.256386 441 leader_election.cc:302] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c63cd279a6bc4a36979d5fce9907aa50; no voters:
I20260619 07:02:20.256765 441 leader_election.cc:288] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:20.256898 451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:20.257176 451 raft_consensus.cc:686] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [term 1 LEADER]: Becoming Leader. State: Replica: c63cd279a6bc4a36979d5fce9907aa50, State: Running, Role: LEADER
I20260619 07:02:20.257576 451 consensus_queue.cc:237] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63cd279a6bc4a36979d5fce9907aa50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } }
I20260619 07:02:20.257946 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:20.255888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:59263 (local address 127.0.0.1:40497)
0619 07:02:20.256023 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:20.256027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:20.256280 (+ 253us) server_negotiation.cc:407] Connection header received
0619 07:02:20.256366 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:20.256369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:20.256404 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:20.256488 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:20.257393 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:20.257397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:20.257399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:20.257411 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:20.257438 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:20.257450 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:20.257504 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:20.257727 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 07:02:20.257782 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 07:02:20.258158 441 sys_catalog.cc:564] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:20.259579 453 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63cd279a6bc4a36979d5fce9907aa50. Latest consensus state: current_term: 1 leader_uuid: "c63cd279a6bc4a36979d5fce9907aa50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63cd279a6bc4a36979d5fce9907aa50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260619 07:02:20.259725 453 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:20.260082 452 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c63cd279a6bc4a36979d5fce9907aa50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63cd279a6bc4a36979d5fce9907aa50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40497 } } }
I20260619 07:02:20.260249 452 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50 [sys.catalog]: This master's current role is: LEADER
W20260619 07:02:20.263233 466 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:20.263339 466 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:20.263443 463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:20.264245 463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:20.269953 463 catalog_manager.cc:1347] Generated new cluster ID: 45a280dc35b04631b7b31fa12960145c
I20260619 07:02:20.270023 463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:20.281351 463 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:20.281692 463 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:20.288362 463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c63cd279a6bc4a36979d5fce9907aa50: Generated new TSK 0
I20260619 07:02:20.289157 463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:20.297598 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 371
2026-06-19T07:02:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:20 dist-test-slave-10d0 krb5kdc[472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:20 dist-test-slave-10d0 krb5kdc[472](info): set up 2 sockets
Jun 19 07:02:20 dist-test-slave-10d0 krb5kdc[472](info): commencing operation
krb5kdc: starting...
W20260619 07:02:21.365374 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:21 dist-test-slave-10d0 krb5kdc[472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:21.489753 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:21.599704 485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:21.599957 485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:21.599998 485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:21.603837 485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:21.603888 485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:21.603906 485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:21.603922 485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:21.603937 485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:21.609134 485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:21.610124 485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:21.611104 485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:21.619472 490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:21.619630 485 server_base.cc:1034] running on GCE node
W20260619 07:02:21.619483 491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:21.619565 493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:21.620303 485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:21.621476 485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:21.622648 485 hybrid_clock.cc:648] HybridClock initialized: now 1781852541622618 us; error 40 us; skew 500 ppm
Jun 19 07:02:21 dist-test-slave-10d0 krb5kdc[472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:21.625428 485 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:21.626307 485 webserver.cc:460] Webserver started at http://127.0.0.1:40173/ using document root <none> and password file <none>
I20260619 07:02:21.626864 485 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:21.626950 485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:21.627157 485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:21.629218 485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "5dbdf83765404100b2e724f23416d7e2"
format_stamp: "Formatted at 2026-06-19 07:02:21 on dist-test-slave-10d0"
server_key: "849cf1fdd4f20324a032c8b881d4a07a"
server_key_iv: "fc112902ae60cf9c3faabe612a36ff13"
server_key_version: "encryptionkey@0"
I20260619 07:02:21.629762 485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "5dbdf83765404100b2e724f23416d7e2"
format_stamp: "Formatted at 2026-06-19 07:02:21 on dist-test-slave-10d0"
server_key: "849cf1fdd4f20324a032c8b881d4a07a"
server_key_iv: "fc112902ae60cf9c3faabe612a36ff13"
server_key_version: "encryptionkey@0"
I20260619 07:02:21.633352 485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:02:21.635749 500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:21.637029 485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:21.637172 485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "5dbdf83765404100b2e724f23416d7e2"
format_stamp: "Formatted at 2026-06-19 07:02:21 on dist-test-slave-10d0"
server_key: "849cf1fdd4f20324a032c8b881d4a07a"
server_key_iv: "fc112902ae60cf9c3faabe612a36ff13"
server_key_version: "encryptionkey@0"
I20260619 07:02:21.637454 485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:21.678596 485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:21.682058 485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:21.682265 485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:21.689422 485 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45023
I20260619 07:02:21.689450 553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45023 every 8 connection(s)
I20260619 07:02:21.690379 485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:21.693187 554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:21.697336 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 485
I20260619 07:02:21.697485 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:21.697731 11689 external_mini_cluster.cc:1428] Setting key aeb6dbd7fed8290e8a18e292abfe8a50
I20260619 07:02:21.699014 554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2: Bootstrap starting.
I20260619 07:02:21.701807 554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:21.702703 554 log.cc:824] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:21.704780 554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2: No bootstrap required, opened a new log
Jun 19 07:02:21 dist-test-slave-10d0 krb5kdc[472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 07:02:21.709139 554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbdf83765404100b2e724f23416d7e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } }
I20260619 07:02:21.709424 554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:21.709489 554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dbdf83765404100b2e724f23416d7e2, State: Initialized, Role: FOLLOWER
I20260619 07:02:21.710022 554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbdf83765404100b2e724f23416d7e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } }
I20260619 07:02:21.710186 554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:21.710268 554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:21.710371 554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:21.711568 554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbdf83765404100b2e724f23416d7e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } }
I20260619 07:02:21.711953 554 leader_election.cc:302] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dbdf83765404100b2e724f23416d7e2; no voters:
I20260619 07:02:21.712342 554 leader_election.cc:288] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:21.712723 559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:21.712918 559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [term 1 LEADER]: Becoming Leader. State: Replica: 5dbdf83765404100b2e724f23416d7e2, State: Running, Role: LEADER
I20260619 07:02:21.713250 559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbdf83765404100b2e724f23416d7e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } }
I20260619 07:02:21.713449 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:21.699199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:45163 (local address 127.0.0.1:45023)
0619 07:02:21.699609 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:21.699628 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:21.699670 (+ 42us) server_negotiation.cc:407] Connection header received
0619 07:02:21.700519 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:21.700572 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:21.701004 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:21.701371 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:21.702149 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:21.703497 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:21.704350 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:21.705050 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:21.705233 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:21.707905 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:21.707912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:21.707929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:21.707961 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:21.709943 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:21.710636 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:21.710661 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:21.710667 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:21.710808 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:21.711143 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:21.711147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:21.711149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:21.711708 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:21.711901 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:21.712201 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 07:02:21.712443 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":103,"threads_started":1}
I20260619 07:02:21.713693 554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:21.715005 561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dbdf83765404100b2e724f23416d7e2. Latest consensus state: current_term: 1 leader_uuid: "5dbdf83765404100b2e724f23416d7e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbdf83765404100b2e724f23416d7e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } } }
I20260619 07:02:21.715159 561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:21.715005 560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dbdf83765404100b2e724f23416d7e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dbdf83765404100b2e724f23416d7e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } } }
I20260619 07:02:21.715245 560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:21.716372 572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:21.716790 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:21.719952 572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:21.725862 572 catalog_manager.cc:1347] Generated new cluster ID: 6721e20cb52342909b2bdb9a73be6842
I20260619 07:02:21.725925 572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:21.727918 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:21.718988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:48033 (local address 127.0.0.1:45023)
0619 07:02:21.719115 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:21.719119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:21.719473 (+ 354us) server_negotiation.cc:407] Connection header received
0619 07:02:21.719580 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:21.719583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:21.719632 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:21.719719 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:21.720521 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:21.721369 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:21.722328 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:21.722579 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:21.722695 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:21.724193 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:21.724197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:21.724200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:21.724227 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:21.726072 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:21.726607 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:21.726611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:21.726613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:21.726703 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:21.727035 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:21.727038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:21.727040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:21.727248 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:21.727387 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:21.727661 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 07:02:21.727766 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 07:02:21.749445 572 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:21.750720 572 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:21.756979 572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dbdf83765404100b2e724f23416d7e2: Generated new TSK 0
I20260619 07:02:21.757766 572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:21.764050 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 485
W20260619 07:02:21.770103 470 connection.cc:535] client connection to 127.0.0.1:45023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45023): Connection reset by peer (error 104)
2026-06-19T07:02:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:21 dist-test-slave-10d0 krb5kdc[585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:21 dist-test-slave-10d0 krb5kdc[585](info): set up 2 sockets
Jun 19 07:02:21 dist-test-slave-10d0 krb5kdc[585](info): commencing operation
krb5kdc: starting...
W20260619 07:02:22.823210 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:22 dist-test-slave-10d0 krb5kdc[585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:22.947319 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:23.057096 598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:23.057346 598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:23.057386 598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:23.061311 598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:23.061371 598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:23.061391 598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:23.061407 598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:23.061422 598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:23.066711 598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:23.067723 598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:23.068748 598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:23.077024 603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:23.077085 598 server_base.cc:1034] running on GCE node
W20260619 07:02:23.077036 604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:23.077033 606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:23.077812 598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:23.078922 598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:23.080103 598 hybrid_clock.cc:648] HybridClock initialized: now 1781852543080075 us; error 49 us; skew 500 ppm
Jun 19 07:02:23 dist-test-slave-10d0 krb5kdc[585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:23.082911 598 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:23.083865 598 webserver.cc:460] Webserver started at http://127.0.0.1:36495/ using document root <none> and password file <none>
I20260619 07:02:23.084455 598 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:23.084518 598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:23.084803 598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:23.086879 598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "0ad776f0fb114d71ad766e75024bacf0"
format_stamp: "Formatted at 2026-06-19 07:02:23 on dist-test-slave-10d0"
server_key: "69cd85e2fd052ce61b78f0682b2cd116"
server_key_iv: "0e12b376062c201e8b7b9cf3164db7bf"
server_key_version: "encryptionkey@0"
I20260619 07:02:23.087435 598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "0ad776f0fb114d71ad766e75024bacf0"
format_stamp: "Formatted at 2026-06-19 07:02:23 on dist-test-slave-10d0"
server_key: "69cd85e2fd052ce61b78f0682b2cd116"
server_key_iv: "0e12b376062c201e8b7b9cf3164db7bf"
server_key_version: "encryptionkey@0"
I20260619 07:02:23.091239 598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 07:02:23.093663 613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:23.094838 598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:23.094967 598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "0ad776f0fb114d71ad766e75024bacf0"
format_stamp: "Formatted at 2026-06-19 07:02:23 on dist-test-slave-10d0"
server_key: "69cd85e2fd052ce61b78f0682b2cd116"
server_key_iv: "0e12b376062c201e8b7b9cf3164db7bf"
server_key_version: "encryptionkey@0"
I20260619 07:02:23.095065 598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:23.124258 598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:23.127451 598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:23.127714 598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:23.135517 598 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42277
I20260619 07:02:23.135547 666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42277 every 8 connection(s)
I20260619 07:02:23.136520 598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:23.139235 667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:23.143208 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 598
I20260619 07:02:23.143353 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:23.143663 11689 external_mini_cluster.cc:1428] Setting key 43e7afc8d72f06cc3152da420106fb3c
I20260619 07:02:23.145239 667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0: Bootstrap starting.
I20260619 07:02:23.147467 667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:23.148214 667 log.cc:824] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:23.150044 667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0: No bootstrap required, opened a new log
I20260619 07:02:23.153083 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:23.145205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:57041 (local address 127.0.0.1:42277)
0619 07:02:23.145638 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:23.145654 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:23.145693 (+ 39us) server_negotiation.cc:407] Connection header received
0619 07:02:23.146354 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:23.146376 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:23.146760 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:23.147118 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:23.147545 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:23.148705 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:23.149645 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:23.150155 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:23.150347 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:23.151291 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:23.151296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:23.151310 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:23.151715 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:23.151773 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:23.151799 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:23.151964 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:23.152158 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 07:02:23.152302 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260619 07:02:23.153841 667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad776f0fb114d71ad766e75024bacf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260619 07:02:23.154095 667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:23.154160 667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad776f0fb114d71ad766e75024bacf0, State: Initialized, Role: FOLLOWER
I20260619 07:02:23.154582 667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad776f0fb114d71ad766e75024bacf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260619 07:02:23.154731 667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:23.154812 667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:23.154919 667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:23.155913 667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad776f0fb114d71ad766e75024bacf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260619 07:02:23.156184 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:23.156306 667 leader_election.cc:302] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ad776f0fb114d71ad766e75024bacf0; no voters:
I20260619 07:02:23.156642 667 leader_election.cc:288] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:23.156736 672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:23.156967 672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [term 1 LEADER]: Becoming Leader. State: Replica: 0ad776f0fb114d71ad766e75024bacf0, State: Running, Role: LEADER
I20260619 07:02:23.157323 672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad776f0fb114d71ad766e75024bacf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } }
I20260619 07:02:23.158046 667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:23.159062 674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ad776f0fb114d71ad766e75024bacf0. Latest consensus state: current_term: 1 leader_uuid: "0ad776f0fb114d71ad766e75024bacf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad776f0fb114d71ad766e75024bacf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } } }
I20260619 07:02:23.159192 674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:23.159513 673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ad776f0fb114d71ad766e75024bacf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad776f0fb114d71ad766e75024bacf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42277 } } }
I20260619 07:02:23.159631 673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0 [sys.catalog]: This master's current role is: LEADER
W20260619 07:02:23.163244 692 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:23.163457 692 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:23.163589 686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:23.164103 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:23.158873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:57459 (local address 127.0.0.1:42277)
0619 07:02:23.158991 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:23.158996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:23.160265 (+ 1269us) server_negotiation.cc:407] Connection header received
0619 07:02:23.160455 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:23.160459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:23.160507 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:23.160613 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:23.161026 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:23.161539 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:23.162391 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:23.162625 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:23.162731 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:23.163581 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:23.163595 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:23.163599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:23.163616 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:23.163645 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:23.163660 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:23.163748 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:23.163939 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 07:02:23.163975 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 07:02:23.164613 686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:23.169911 686 catalog_manager.cc:1347] Generated new cluster ID: efb6be71871644d0b4808c5b0b99cd31
I20260619 07:02:23.169981 686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:23.194034 686 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:23.195178 686 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:23.200881 686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ad776f0fb114d71ad766e75024bacf0: Generated new TSK 0
I20260619 07:02:23.201629 686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:23.209954 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 598
W20260619 07:02:23.215371 583 connection.cc:535] client connection to 127.0.0.1:42277 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42277): Connection reset by peer (error 104)
2026-06-19T07:02:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:23 dist-test-slave-10d0 krb5kdc[698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:23 dist-test-slave-10d0 krb5kdc[698](info): set up 2 sockets
Jun 19 07:02:23 dist-test-slave-10d0 krb5kdc[698](info): commencing operation
krb5kdc: starting...
W20260619 07:02:24.257229 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:24 dist-test-slave-10d0 krb5kdc[698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 07:02:24.375900 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:24.485914 711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:24.486260 711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:24.486349 711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:24.490432 711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:24.490533 711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:24.490573 711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:24.490628 711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:24.490662 711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:24.496097 711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:24.497246 711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:24.498318 711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:24.506718 717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:24.506910 711 server_base.cc:1034] running on GCE node
W20260619 07:02:24.506717 719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:24.506780 716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:24.507754 711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:24.508934 711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:24.510126 711 hybrid_clock.cc:648] HybridClock initialized: now 1781852544510113 us; error 26 us; skew 500 ppm
Jun 19 07:02:24 dist-test-slave-10d0 krb5kdc[698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:24.517803 711 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 07:02:24.518748 711 webserver.cc:460] Webserver started at http://127.0.0.1:33827/ using document root <none> and password file <none>
I20260619 07:02:24.519294 711 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:24.519353 711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:24.519523 711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:24.521601 711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "3e935fa59f8a4d74b783fc5d05401c0f"
format_stamp: "Formatted at 2026-06-19 07:02:24 on dist-test-slave-10d0"
server_key: "256ef1407ea500cde2d9da0a4a660627"
server_key_iv: "6aa271b32f2fea5ea67412ad5e0f6d1f"
server_key_version: "encryptionkey@0"
I20260619 07:02:24.522107 711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "3e935fa59f8a4d74b783fc5d05401c0f"
format_stamp: "Formatted at 2026-06-19 07:02:24 on dist-test-slave-10d0"
server_key: "256ef1407ea500cde2d9da0a4a660627"
server_key_iv: "6aa271b32f2fea5ea67412ad5e0f6d1f"
server_key_version: "encryptionkey@0"
I20260619 07:02:24.525770 711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:02:24.528077 726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:24.529312 711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:24.529441 711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "3e935fa59f8a4d74b783fc5d05401c0f"
format_stamp: "Formatted at 2026-06-19 07:02:24 on dist-test-slave-10d0"
server_key: "256ef1407ea500cde2d9da0a4a660627"
server_key_iv: "6aa271b32f2fea5ea67412ad5e0f6d1f"
server_key_version: "encryptionkey@0"
I20260619 07:02:24.529538 711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:24.534137 711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:24.537002 711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:24.537191 711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:24.544510 711 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46577
I20260619 07:02:24.544519 778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46577 every 8 connection(s)
I20260619 07:02:24.545497 711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:24.548206 779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:24.552027 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 711
I20260619 07:02:24.552197 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:24.552490 11689 external_mini_cluster.cc:1428] Setting key 0f44db6a548f2ae7c8f3f020604c2c0d
I20260619 07:02:24.553961 779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f: Bootstrap starting.
I20260619 07:02:24.556411 779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:24.557205 779 log.cc:824] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:24.557493 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:24.553773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:59899 (local address 127.0.0.1:46577)
0619 07:02:24.554183 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:24.554198 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:24.554234 (+ 36us) server_negotiation.cc:407] Connection header received
0619 07:02:24.554856 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:24.554873 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:24.555150 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:24.555424 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:24.556200 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:24.556205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:24.556217 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:24.556226 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:24.556275 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:24.556293 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:24.556375 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:24.556624 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 07:02:24.556795 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":128,"threads_started":1}
I20260619 07:02:24.559229 779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f: No bootstrap required, opened a new log
I20260619 07:02:24.560261 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:24.564038 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:24.562507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:60051 (local address 127.0.0.1:46577)
0619 07:02:24.562655 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:24.562661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:24.562784 (+ 123us) server_negotiation.cc:407] Connection header received
0619 07:02:24.562861 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:24.562866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:24.562907 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:24.562990 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:24.563661 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:24.563665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:24.563668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:24.563679 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:24.563706 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:24.563718 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:24.563805 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:24.563857 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 07:02:24.563891 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 07:02:24.563930 779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e935fa59f8a4d74b783fc5d05401c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46577 } }
I20260619 07:02:24.564230 779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:24.564294 779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e935fa59f8a4d74b783fc5d05401c0f, State: Initialized, Role: FOLLOWER
I20260619 07:02:24.564772 779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e935fa59f8a4d74b783fc5d05401c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46577 } }
I20260619 07:02:24.564932 779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:24.565002 779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:24.565119 779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:24.565915 779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e935fa59f8a4d74b783fc5d05401c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46577 } }
I20260619 07:02:24.566205 779 leader_election.cc:302] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e935fa59f8a4d74b783fc5d05401c0f; no voters:
I20260619 07:02:24.566432 779 leader_election.cc:288] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:24.566541 790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:24.566779 790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [term 1 LEADER]: Becoming Leader. State: Replica: 3e935fa59f8a4d74b783fc5d05401c0f, State: Running, Role: LEADER
I20260619 07:02:24.567080 790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e935fa59f8a4d74b783fc5d05401c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46577 } }
I20260619 07:02:24.567565 779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:24.568168 790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e935fa59f8a4d74b783fc5d05401c0f. Latest consensus state: current_term: 1 leader_uuid: "3e935fa59f8a4d74b783fc5d05401c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e935fa59f8a4d74b783fc5d05401c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46577 } } }
I20260619 07:02:24.568338 790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:24.568631 792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e935fa59f8a4d74b783fc5d05401c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e935fa59f8a4d74b783fc5d05401c0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46577 } } }
I20260619 07:02:24.568795 792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:24.568787 799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:24.571873 799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:24.577519 799 catalog_manager.cc:1347] Generated new cluster ID: a3804b7744124eb3b77b5437ec242370
I20260619 07:02:24.577589 799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:24.585561 799 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:24.585855 799 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:24.591130 799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e935fa59f8a4d74b783fc5d05401c0f: Generated new TSK 0
I20260619 07:02:24.591753 799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:24.599550 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 711
2026-06-19T07:02:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:24 dist-test-slave-10d0 krb5kdc[811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:24 dist-test-slave-10d0 krb5kdc[811](info): set up 2 sockets
Jun 19 07:02:24 dist-test-slave-10d0 krb5kdc[811](info): commencing operation
krb5kdc: starting...
W20260619 07:02:25.633405 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:25 dist-test-slave-10d0 krb5kdc[811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:25.753712 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:35697
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:35697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:25.864678 824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:25.864959 824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:25.865031 824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:25.869053 824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:25.869120 824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:25.869140 824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:25.869158 824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:25.869215 824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:25.874490 824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:35697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:35697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:25.875581 824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:25.876654 824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:25.885565 830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:25.885532 832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:25.885527 829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:25.885942 824 server_base.cc:1034] running on GCE node
I20260619 07:02:25.886507 824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:25.887625 824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:25.888789 824 hybrid_clock.cc:648] HybridClock initialized: now 1781852545888770 us; error 31 us; skew 500 ppm
Jun 19 07:02:25 dist-test-slave-10d0 krb5kdc[811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852545, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:25.891389 824 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:25.892236 824 webserver.cc:460] Webserver started at http://127.11.106.126:46585/ using document root <none> and password file <none>
I20260619 07:02:25.892855 824 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:25.892942 824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:25.893165 824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:25.895167 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "45dc632daf554ac4b9af4805a981c461"
format_stamp: "Formatted at 2026-06-19 07:02:25 on dist-test-slave-10d0"
server_key: "d9d37e27268aa76afec4dccf33bca49e"
server_key_iv: "e1ce3ba5b81ca9c6523eebf302cf26cc"
server_key_version: "encryptionkey@0"
I20260619 07:02:25.895732 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "45dc632daf554ac4b9af4805a981c461"
format_stamp: "Formatted at 2026-06-19 07:02:25 on dist-test-slave-10d0"
server_key: "d9d37e27268aa76afec4dccf33bca49e"
server_key_iv: "e1ce3ba5b81ca9c6523eebf302cf26cc"
server_key_version: "encryptionkey@0"
I20260619 07:02:25.899483 824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 07:02:25.901768 839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:25.902958 824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:25.903096 824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "45dc632daf554ac4b9af4805a981c461"
format_stamp: "Formatted at 2026-06-19 07:02:25 on dist-test-slave-10d0"
server_key: "d9d37e27268aa76afec4dccf33bca49e"
server_key_iv: "e1ce3ba5b81ca9c6523eebf302cf26cc"
server_key_version: "encryptionkey@0"
I20260619 07:02:25.903204 824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:25.919466 824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:25.922293 824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:25.922490 824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:25.929087 824 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:35697
I20260619 07:02:25.929116 891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:35697 every 8 connection(s)
I20260619 07:02:25.930007 824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:25.932807 892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:25.938066 892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461: Bootstrap starting.
I20260619 07:02:25.939570 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 824
I20260619 07:02:25.939713 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:25.940011 11689 external_mini_cluster.cc:1428] Setting key f3f9540d0ca08d40d4eef6e519968eb4
I20260619 07:02:25.940764 892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:25.941619 892 log.cc:824] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:25.943567 892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461: No bootstrap required, opened a new log
I20260619 07:02:25.946883 892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dc632daf554ac4b9af4805a981c461" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35697 } }
I20260619 07:02:25.947120 892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:25.947196 892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dc632daf554ac4b9af4805a981c461, State: Initialized, Role: FOLLOWER
I20260619 07:02:25.947613 892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dc632daf554ac4b9af4805a981c461" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35697 } }
I20260619 07:02:25.947757 892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:25.947836 892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:25.947957 892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 07:02:25 dist-test-slave-10d0 krb5kdc[811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:25.949167 892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dc632daf554ac4b9af4805a981c461" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35697 } }
I20260619 07:02:25.949534 892 leader_election.cc:302] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45dc632daf554ac4b9af4805a981c461; no voters:
I20260619 07:02:25.949844 892 leader_election.cc:288] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:25.950165 897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:25.950434 897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [term 1 LEADER]: Becoming Leader. State: Replica: 45dc632daf554ac4b9af4805a981c461, State: Running, Role: LEADER
I20260619 07:02:25.950803 897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dc632daf554ac4b9af4805a981c461" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35697 } }
I20260619 07:02:25.951124 892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:25.952808 899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45dc632daf554ac4b9af4805a981c461. Latest consensus state: current_term: 1 leader_uuid: "45dc632daf554ac4b9af4805a981c461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dc632daf554ac4b9af4805a981c461" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35697 } } }
I20260619 07:02:25.952945 899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:25.953269 898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45dc632daf554ac4b9af4805a981c461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45dc632daf554ac4b9af4805a981c461" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 35697 } } }
I20260619 07:02:25.954437 898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:25.955209 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:25.941145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44263 (local address 127.11.106.126:35697)
0619 07:02:25.941549 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:25.941564 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:25.941610 (+ 46us) server_negotiation.cc:407] Connection header received
0619 07:02:25.942439 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:25.942458 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:25.942802 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:25.943119 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:25.943945 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:25.944947 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:25.945806 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:25.946282 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:25.946450 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:25.949464 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:25.949470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:25.949482 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:25.949508 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:25.951536 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:25.952174 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:25.952180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:25.952198 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:25.952348 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:25.952888 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:25.952895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:25.952899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:25.953375 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:25.953596 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:25.954230 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 07:02:25.954478 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
W20260619 07:02:25.957756 912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:25.957834 912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:25.957894 911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:25.958209 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:25.958791 911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:25.964834 911 catalog_manager.cc:1347] Generated new cluster ID: 84272ddca88840bb957b62bf4899ae17
I20260619 07:02:25.964900 911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:25.969138 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:25.960686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57559 (local address 127.11.106.126:35697)
0619 07:02:25.960786 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:25.960789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:25.960802 (+ 13us) server_negotiation.cc:407] Connection header received
0619 07:02:25.960837 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:25.960839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:25.960878 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:25.960961 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:25.961993 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:25.962665 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:25.963610 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:25.963851 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:25.963954 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:25.965589 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:25.965593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:25.965596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:25.965623 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:25.967313 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:25.967818 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:25.967821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:25.967823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:25.967910 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:25.968246 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:25.968249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:25.968251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:25.968470 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:25.968643 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:25.968886 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 07:02:25.968985 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 07:02:25.987926 911 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:25.989045 911 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:25.998625 911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45dc632daf554ac4b9af4805a981c461: Generated new TSK 0
I20260619 07:02:25.999312 911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:26.006243 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 824
W20260619 07:02:26.011970 809 connection.cc:535] client connection to 127.11.106.126:35697 recv error: Network error: failed to read from TLS socket (remote: 127.11.106.126:35697): Connection reset by peer (error 104)
2026-06-19T07:02:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:26 dist-test-slave-10d0 krb5kdc[923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:26 dist-test-slave-10d0 krb5kdc[923](info): set up 2 sockets
Jun 19 07:02:26 dist-test-slave-10d0 krb5kdc[923](info): commencing operation
krb5kdc: starting...
W20260619 07:02:27.085750 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:27 dist-test-slave-10d0 krb5kdc[923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:27.207726 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:43369
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:43369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:27.317150 936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:27.317457 936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:27.317549 936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:27.321666 936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:27.321750 936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:27.321789 936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:27.321846 936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:27.321885 936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:27.327235 936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:43369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:43369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:27.328286 936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:27.329371 936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:27.337764 941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:27.337736 942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:27.337746 944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:27.338281 936 server_base.cc:1034] running on GCE node
I20260619 07:02:27.338757 936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:27.339814 936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:27.340972 936 hybrid_clock.cc:648] HybridClock initialized: now 1781852547340950 us; error 34 us; skew 500 ppm
Jun 19 07:02:27 dist-test-slave-10d0 krb5kdc[923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852547, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:27.343660 936 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:27.344509 936 webserver.cc:460] Webserver started at http://127.11.106.126:41181/ using document root <none> and password file <none>
I20260619 07:02:27.345077 936 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:27.345140 936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:27.345297 936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:27.347205 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "450229c3f6d6407fa9d6ff3c0462ca1e"
format_stamp: "Formatted at 2026-06-19 07:02:27 on dist-test-slave-10d0"
server_key: "f77e0fea7381456b397ccaf38f12e4a1"
server_key_iv: "f1b7bad95bd5d326bdd6d0e26334efd8"
server_key_version: "encryptionkey@0"
I20260619 07:02:27.347689 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "450229c3f6d6407fa9d6ff3c0462ca1e"
format_stamp: "Formatted at 2026-06-19 07:02:27 on dist-test-slave-10d0"
server_key: "f77e0fea7381456b397ccaf38f12e4a1"
server_key_iv: "f1b7bad95bd5d326bdd6d0e26334efd8"
server_key_version: "encryptionkey@0"
I20260619 07:02:27.351254 936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 07:02:27.353760 951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:27.354940 936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:27.355085 936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "450229c3f6d6407fa9d6ff3c0462ca1e"
format_stamp: "Formatted at 2026-06-19 07:02:27 on dist-test-slave-10d0"
server_key: "f77e0fea7381456b397ccaf38f12e4a1"
server_key_iv: "f1b7bad95bd5d326bdd6d0e26334efd8"
server_key_version: "encryptionkey@0"
I20260619 07:02:27.355195 936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:27.382741 936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:27.386112 936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:27.386327 936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:27.393651 936 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:43369
I20260619 07:02:27.393668 1003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:43369 every 8 connection(s)
I20260619 07:02:27.394634 936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:27.397327 1004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:27.402884 1004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e: Bootstrap starting.
I20260619 07:02:27.404477 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 936
I20260619 07:02:27.404644 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:27.404948 11689 external_mini_cluster.cc:1428] Setting key dd5425c059ab6f411356e0d9a538ce8b
I20260619 07:02:27.405619 1004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:27.406545 1004 log.cc:824] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:27.408391 1004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e: No bootstrap required, opened a new log
I20260619 07:02:27.411762 1004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450229c3f6d6407fa9d6ff3c0462ca1e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43369 } }
I20260619 07:02:27.411998 1004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:27.412094 1004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450229c3f6d6407fa9d6ff3c0462ca1e, State: Initialized, Role: FOLLOWER
I20260619 07:02:27.412526 1004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450229c3f6d6407fa9d6ff3c0462ca1e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43369 } }
I20260619 07:02:27.412731 1004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:27.412819 1004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:27.412921 1004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:27.413859 1004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450229c3f6d6407fa9d6ff3c0462ca1e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43369 } }
I20260619 07:02:27.413908 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:27.406096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36287 (local address 127.11.106.126:43369)
0619 07:02:27.406524 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:27.406539 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:27.406574 (+ 35us) server_negotiation.cc:407] Connection header received
0619 07:02:27.407419 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:27.407439 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:27.407775 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:27.408104 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:27.408532 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:27.409540 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:27.410467 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:27.410941 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:27.411112 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:27.412045 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:27.412052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:27.412077 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:27.412673 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:27.412752 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:27.412784 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:27.412915 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:27.413094 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 07:02:27.413233 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260619 07:02:27.414278 1004 leader_election.cc:302] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 450229c3f6d6407fa9d6ff3c0462ca1e; no voters:
I20260619 07:02:27.414602 1004 leader_election.cc:288] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:27.414705 1009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:27.414912 1009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [term 1 LEADER]: Becoming Leader. State: Replica: 450229c3f6d6407fa9d6ff3c0462ca1e, State: Running, Role: LEADER
I20260619 07:02:27.415256 1009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450229c3f6d6407fa9d6ff3c0462ca1e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43369 } }
I20260619 07:02:27.415825 1004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:27.416519 1011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 450229c3f6d6407fa9d6ff3c0462ca1e. Latest consensus state: current_term: 1 leader_uuid: "450229c3f6d6407fa9d6ff3c0462ca1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450229c3f6d6407fa9d6ff3c0462ca1e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43369 } } }
I20260619 07:02:27.417269 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:27.417335 1011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:27.417224 1010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "450229c3f6d6407fa9d6ff3c0462ca1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450229c3f6d6407fa9d6ff3c0462ca1e" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 43369 } } }
I20260619 07:02:27.417557 1010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:27.418493 1019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:27.421850 1019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:27.423506 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:27.419461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52723 (local address 127.11.106.126:43369)
0619 07:02:27.419572 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:27.419587 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:27.419795 (+ 208us) server_negotiation.cc:407] Connection header received
0619 07:02:27.419939 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:27.419942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:27.419983 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:27.420074 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:27.420336 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:27.420922 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:27.421751 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:27.422068 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:27.422182 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:27.423027 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:27.423030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:27.423033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:27.423045 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:27.423078 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:27.423096 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:27.423196 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:27.423320 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 07:02:27.423354 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 07:02:27.427758 1019 catalog_manager.cc:1347] Generated new cluster ID: 3d75f25df37549a0a863a53992f41f8d
I20260619 07:02:27.427830 1019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:27.438628 1019 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:27.439815 1019 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:27.446723 1019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 450229c3f6d6407fa9d6ff3c0462ca1e: Generated new TSK 0
I20260619 07:02:27.447479 1019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:27.451298 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 936
W20260619 07:02:27.456364 921 connection.cc:535] client connection to 127.11.106.126:43369 recv error: Network error: failed to read from TLS socket (remote: 127.11.106.126:43369): Connection reset by peer (error 104)
2026-06-19T07:02:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:27 dist-test-slave-10d0 krb5kdc[1035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:27 dist-test-slave-10d0 krb5kdc[1035](info): set up 2 sockets
Jun 19 07:02:27 dist-test-slave-10d0 krb5kdc[1035](info): commencing operation
krb5kdc: starting...
W20260619 07:02:28.505442 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:28 dist-test-slave-10d0 krb5kdc[1035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:28.629761 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:33315
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:33315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:28.742980 1049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:28.743283 1049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:28.743326 1049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:28.747375 1049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:28.747445 1049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:28.747465 1049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:28.747483 1049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:28.747498 1049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:28.752980 1049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:33315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:33315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:28.754086 1049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:28.755139 1049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:28.764113 1054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:28.764138 1057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:28.764371 1049 server_base.cc:1034] running on GCE node
W20260619 07:02:28.764113 1055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:28.765122 1049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:28.766317 1049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:28.767514 1049 hybrid_clock.cc:648] HybridClock initialized: now 1781852548767495 us; error 33 us; skew 500 ppm
Jun 19 07:02:28 dist-test-slave-10d0 krb5kdc[1035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852548, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:28.770398 1049 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:28.771389 1049 webserver.cc:460] Webserver started at http://127.11.106.126:45351/ using document root <none> and password file <none>
I20260619 07:02:28.771988 1049 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:28.772082 1049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:28.772320 1049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:28.774482 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "6850345dd17f4808a3fdbf45141d803b"
format_stamp: "Formatted at 2026-06-19 07:02:28 on dist-test-slave-10d0"
server_key: "ef1a7f3ad8f00c1a554d76b8481e96e6"
server_key_iv: "b45212516fbb503bca1916615c655dd4"
server_key_version: "encryptionkey@0"
I20260619 07:02:28.775072 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "6850345dd17f4808a3fdbf45141d803b"
format_stamp: "Formatted at 2026-06-19 07:02:28 on dist-test-slave-10d0"
server_key: "ef1a7f3ad8f00c1a554d76b8481e96e6"
server_key_iv: "b45212516fbb503bca1916615c655dd4"
server_key_version: "encryptionkey@0"
I20260619 07:02:28.778997 1049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 07:02:28.781538 1064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:28.782851 1049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:28.782997 1049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "6850345dd17f4808a3fdbf45141d803b"
format_stamp: "Formatted at 2026-06-19 07:02:28 on dist-test-slave-10d0"
server_key: "ef1a7f3ad8f00c1a554d76b8481e96e6"
server_key_iv: "b45212516fbb503bca1916615c655dd4"
server_key_version: "encryptionkey@0"
I20260619 07:02:28.783121 1049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:28.787870 1049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:28.790798 1049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:28.790993 1049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:28.798722 1049 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:33315
I20260619 07:02:28.798751 1116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:33315 every 8 connection(s)
I20260619 07:02:28.799700 1049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:28.802594 1117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:28.806246 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 1049
I20260619 07:02:28.806411 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:28.806712 11689 external_mini_cluster.cc:1428] Setting key c5305510f2da26307f675c926234bccc
I20260619 07:02:28.809011 1117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b: Bootstrap starting.
I20260619 07:02:28.811400 1117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:28.812129 1117 log.cc:824] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:28.812163 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:28.808222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40605 (local address 127.11.106.126:33315)
0619 07:02:28.808744 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:28.808760 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:28.808797 (+ 37us) server_negotiation.cc:407] Connection header received
0619 07:02:28.809483 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:28.809502 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:28.809798 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:28.810139 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:28.810935 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:28.810939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:28.810951 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:28.810967 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:28.811031 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:28.811050 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:28.811165 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:28.811272 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 07:02:28.811431 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":122,"threads_started":1}
I20260619 07:02:28.814338 1117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b: No bootstrap required, opened a new log
I20260619 07:02:28.815147 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:28.818979 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:28.817248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48003 (local address 127.11.106.126:33315)
0619 07:02:28.817376 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:28.817380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:28.817636 (+ 256us) server_negotiation.cc:407] Connection header received
0619 07:02:28.817726 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:28.817729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:28.817759 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:28.817831 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:28.818544 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:28.818547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:28.818549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:28.818559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:28.818607 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:28.818617 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:28.818685 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:28.818793 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 07:02:28.818826 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 07:02:28.818784 1117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850345dd17f4808a3fdbf45141d803b" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33315 } }
I20260619 07:02:28.819120 1117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:28.819185 1117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6850345dd17f4808a3fdbf45141d803b, State: Initialized, Role: FOLLOWER
I20260619 07:02:28.819690 1117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850345dd17f4808a3fdbf45141d803b" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33315 } }
I20260619 07:02:28.819859 1117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:28.819969 1117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:28.820096 1117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:28.821134 1117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850345dd17f4808a3fdbf45141d803b" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33315 } }
I20260619 07:02:28.821482 1117 leader_election.cc:302] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6850345dd17f4808a3fdbf45141d803b; no voters:
I20260619 07:02:28.821833 1117 leader_election.cc:288] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:28.821911 1127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:28.822175 1127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [term 1 LEADER]: Becoming Leader. State: Replica: 6850345dd17f4808a3fdbf45141d803b, State: Running, Role: LEADER
I20260619 07:02:28.822551 1127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850345dd17f4808a3fdbf45141d803b" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33315 } }
I20260619 07:02:28.823079 1117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:28.824342 1129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6850345dd17f4808a3fdbf45141d803b. Latest consensus state: current_term: 1 leader_uuid: "6850345dd17f4808a3fdbf45141d803b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850345dd17f4808a3fdbf45141d803b" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33315 } } }
I20260619 07:02:28.824350 1128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6850345dd17f4808a3fdbf45141d803b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850345dd17f4808a3fdbf45141d803b" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 33315 } } }
I20260619 07:02:28.824486 1129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:28.824512 1128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:28.824982 1137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:28.828349 1137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:28.834373 1137 catalog_manager.cc:1347] Generated new cluster ID: d01fb7cff2a6487fbacd760dd5a49f88
I20260619 07:02:28.834462 1137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:28.848455 1137 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:28.848909 1137 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:28.857337 1137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6850345dd17f4808a3fdbf45141d803b: Generated new TSK 0
I20260619 07:02:28.858248 1137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:28.868732 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 1049
2026-06-19T07:02:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:28 dist-test-slave-10d0 krb5kdc[1149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:28 dist-test-slave-10d0 krb5kdc[1149](info): set up 2 sockets
Jun 19 07:02:28 dist-test-slave-10d0 krb5kdc[1149](info): commencing operation
krb5kdc: starting...
W20260619 07:02:29.921401 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:30 dist-test-slave-10d0 krb5kdc[1149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:30.043565 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:37083
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:37083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:30.154393 1162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:30.154688 1162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:30.154731 1162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:30.159483 1162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:30.159549 1162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:30.159570 1162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:30.159618 1162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:30.159649 1162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:30.165220 1162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:37083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:37083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:30.166335 1162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:30.167380 1162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:30.176007 1167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:30.176019 1168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:30.176098 1170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:30.176272 1162 server_base.cc:1034] running on GCE node
I20260619 07:02:30.177026 1162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:30.178187 1162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:30.179374 1162 hybrid_clock.cc:648] HybridClock initialized: now 1781852550179341 us; error 47 us; skew 500 ppm
Jun 19 07:02:30 dist-test-slave-10d0 krb5kdc[1149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852550, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:30.187878 1162 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:30.188906 1162 webserver.cc:460] Webserver started at http://127.11.106.126:41367/ using document root <none> and password file <none>
I20260619 07:02:30.189476 1162 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:30.189575 1162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:30.189810 1162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:30.191856 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "a17b653e53344c5fbc5f552d0b5e645d"
format_stamp: "Formatted at 2026-06-19 07:02:30 on dist-test-slave-10d0"
server_key: "e0f58d6ecf8838fca5799fd4c1a46ed2"
server_key_iv: "a3350b0be66d845fef44510ea1f971e3"
server_key_version: "encryptionkey@0"
I20260619 07:02:30.192404 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "a17b653e53344c5fbc5f552d0b5e645d"
format_stamp: "Formatted at 2026-06-19 07:02:30 on dist-test-slave-10d0"
server_key: "e0f58d6ecf8838fca5799fd4c1a46ed2"
server_key_iv: "a3350b0be66d845fef44510ea1f971e3"
server_key_version: "encryptionkey@0"
I20260619 07:02:30.196041 1162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 07:02:30.198454 1177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:30.199683 1162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:30.199833 1162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "a17b653e53344c5fbc5f552d0b5e645d"
format_stamp: "Formatted at 2026-06-19 07:02:30 on dist-test-slave-10d0"
server_key: "e0f58d6ecf8838fca5799fd4c1a46ed2"
server_key_iv: "a3350b0be66d845fef44510ea1f971e3"
server_key_version: "encryptionkey@0"
I20260619 07:02:30.199945 1162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:30.231319 1162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:30.234836 1162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:30.235046 1162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:30.242628 1162 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:37083
I20260619 07:02:30.242640 1229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:37083 every 8 connection(s)
I20260619 07:02:30.243587 1162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:30.246336 1230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:30.250049 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 1162
I20260619 07:02:30.250214 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:30.250527 11689 external_mini_cluster.cc:1428] Setting key cadfa744e5a212d68f53b5feeb8e44f8
I20260619 07:02:30.252413 1230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d: Bootstrap starting.
I20260619 07:02:30.254590 1230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:30.255326 1230 log.cc:824] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:30.257196 1230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d: No bootstrap required, opened a new log
Jun 19 07:02:30 dist-test-slave-10d0 krb5kdc[1149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:30.261226 1230 raft_consensus.cc:348] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17b653e53344c5fbc5f552d0b5e645d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37083 } }
I20260619 07:02:30.261471 1230 raft_consensus.cc:374] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:30.261550 1230 raft_consensus.cc:729] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17b653e53344c5fbc5f552d0b5e645d, State: Initialized, Role: FOLLOWER
I20260619 07:02:30.262032 1230 consensus_queue.cc:260] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17b653e53344c5fbc5f552d0b5e645d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37083 } }
I20260619 07:02:30.262195 1230 raft_consensus.cc:388] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:30.262283 1230 raft_consensus.cc:482] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:30.262380 1230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:30.263523 1230 raft_consensus.cc:504] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17b653e53344c5fbc5f552d0b5e645d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37083 } }
I20260619 07:02:30.263880 1230 leader_election.cc:302] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a17b653e53344c5fbc5f552d0b5e645d; no voters:
I20260619 07:02:30.264243 1230 leader_election.cc:288] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:30.264331 1235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:30.264748 1235 raft_consensus.cc:686] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [term 1 LEADER]: Becoming Leader. State: Replica: a17b653e53344c5fbc5f552d0b5e645d, State: Running, Role: LEADER
I20260619 07:02:30.265055 1235 consensus_queue.cc:237] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17b653e53344c5fbc5f552d0b5e645d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37083 } }
I20260619 07:02:30.265547 1230 sys_catalog.cc:564] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:30.265857 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:30.251990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47235 (local address 127.11.106.126:37083)
0619 07:02:30.252418 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:30.252434 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:30.252492 (+ 58us) server_negotiation.cc:407] Connection header received
0619 07:02:30.253180 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:30.253196 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:30.253538 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:30.253873 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:30.254674 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:30.255753 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:30.256713 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:30.257181 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:30.257351 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:30.260257 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:30.260263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:30.260277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:30.260311 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:30.262335 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:30.262871 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:30.262878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:30.262883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:30.262990 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:30.263278 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:30.263280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:30.263281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:30.263755 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:30.263897 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:30.264627 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 07:02:30.264916 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":134,"threads_started":1}
I20260619 07:02:30.266172 1235 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a17b653e53344c5fbc5f552d0b5e645d. Latest consensus state: current_term: 1 leader_uuid: "a17b653e53344c5fbc5f552d0b5e645d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17b653e53344c5fbc5f552d0b5e645d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37083 } } }
I20260619 07:02:30.266352 1235 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:30.266631 1236 sys_catalog.cc:455] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a17b653e53344c5fbc5f552d0b5e645d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a17b653e53344c5fbc5f552d0b5e645d" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 37083 } } }
I20260619 07:02:30.266772 1236 sys_catalog.cc:458] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:30.266842 1244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:30.269189 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:30.270083 1244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:30.276031 1244 catalog_manager.cc:1347] Generated new cluster ID: cbdad4ad1b884c94ac1be25b1bbdca30
I20260619 07:02:30.276090 1244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:30.280474 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:30.271324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50181 (local address 127.11.106.126:37083)
0619 07:02:30.271428 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:30.271433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:30.271691 (+ 258us) server_negotiation.cc:407] Connection header received
0619 07:02:30.271791 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:30.271794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:30.271839 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:30.271928 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:30.272799 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:30.273573 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:30.274711 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:30.275072 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:30.275176 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:30.276924 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:30.276927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:30.276930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:30.276953 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:30.278698 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:30.279215 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:30.279218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:30.279220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:30.279299 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:30.279614 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:30.279617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:30.279618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:30.279797 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:30.279904 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:30.280215 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 07:02:30.280319 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 07:02:30.291513 1244 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:30.292484 1244 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:30.297899 1244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a17b653e53344c5fbc5f552d0b5e645d: Generated new TSK 0
I20260619 07:02:30.298609 1244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:30.308588 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 1162
2026-06-19T07:02:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:30 dist-test-slave-10d0 krb5kdc[1261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:30 dist-test-slave-10d0 krb5kdc[1261](info): set up 2 sockets
Jun 19 07:02:30 dist-test-slave-10d0 krb5kdc[1261](info): commencing operation
krb5kdc: starting...
W20260619 07:02:31.357486 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:31 dist-test-slave-10d0 krb5kdc[1261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:31.480319 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42941
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:42941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:31.593227 1275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:31.593482 1275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:31.593521 1275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:31.597483 1275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:31.597535 1275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:31.597554 1275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:31.597571 1275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:31.597635 1275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:31.602758 1275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:42941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:42941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:31.603801 1275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:31.604861 1275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:31.613322 1280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:31.613346 1281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:31.613451 1275 server_base.cc:1034] running on GCE node
W20260619 07:02:31.613327 1283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:31.614174 1275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:31.615242 1275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:31.616430 1275 hybrid_clock.cc:648] HybridClock initialized: now 1781852551616417 us; error 28 us; skew 500 ppm
Jun 19 07:02:31 dist-test-slave-10d0 krb5kdc[1261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852551, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:31.619194 1275 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:31.620045 1275 webserver.cc:460] Webserver started at http://127.11.106.126:41567/ using document root <none> and password file <none>
I20260619 07:02:31.620641 1275 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:31.620707 1275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:31.620878 1275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:31.623239 1275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "54db1b3ba0d94f77aeee1b684c2b905c"
format_stamp: "Formatted at 2026-06-19 07:02:31 on dist-test-slave-10d0"
server_key: "ee54312387fffc043d6bded99ba5745d"
server_key_iv: "9473289a4a400dd2a4eb82b4d3882c2d"
server_key_version: "encryptionkey@0"
I20260619 07:02:31.623775 1275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "54db1b3ba0d94f77aeee1b684c2b905c"
format_stamp: "Formatted at 2026-06-19 07:02:31 on dist-test-slave-10d0"
server_key: "ee54312387fffc043d6bded99ba5745d"
server_key_iv: "9473289a4a400dd2a4eb82b4d3882c2d"
server_key_version: "encryptionkey@0"
I20260619 07:02:31.627620 1275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 07:02:31.630168 1290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:31.631382 1275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:31.631489 1275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "54db1b3ba0d94f77aeee1b684c2b905c"
format_stamp: "Formatted at 2026-06-19 07:02:31 on dist-test-slave-10d0"
server_key: "ee54312387fffc043d6bded99ba5745d"
server_key_iv: "9473289a4a400dd2a4eb82b4d3882c2d"
server_key_version: "encryptionkey@0"
I20260619 07:02:31.631587 1275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:31.653440 1275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:31.656433 1275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:31.656669 1275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:31.663920 1275 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:42941
I20260619 07:02:31.663940 1342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:42941 every 8 connection(s)
I20260619 07:02:31.665009 1275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:31.666764 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 1275
I20260619 07:02:31.666922 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:31.667203 11689 external_mini_cluster.cc:1428] Setting key c47e1b09add5d62e1741f4f3b18f5e77
I20260619 07:02:31.668228 1343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:31.675289 1343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c: Bootstrap starting.
I20260619 07:02:31.677516 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:31.668675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44313 (local address 127.11.106.126:42941)
0619 07:02:31.669103 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:31.669123 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:31.669167 (+ 44us) server_negotiation.cc:407] Connection header received
0619 07:02:31.670024 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:31.670047 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:31.670491 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:31.670864 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:31.671289 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:31.672816 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:31.673706 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:31.674451 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:31.674610 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:31.675459 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:31.675465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:31.675480 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:31.675967 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:31.676033 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:31.676062 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:31.676159 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:31.676310 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 07:02:31.676581 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260619 07:02:31.678112 1343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:31.678975 1343 log.cc:824] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:31.680678 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:31.681030 1343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c: No bootstrap required, opened a new log
I20260619 07:02:31.685133 1343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db1b3ba0d94f77aeee1b684c2b905c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42941 } }
I20260619 07:02:31.685352 1343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:31.685431 1343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54db1b3ba0d94f77aeee1b684c2b905c, State: Initialized, Role: FOLLOWER
I20260619 07:02:31.685925 1343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db1b3ba0d94f77aeee1b684c2b905c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42941 } }
I20260619 07:02:31.686081 1343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:31.686163 1343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:31.686271 1343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:31.687301 1343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db1b3ba0d94f77aeee1b684c2b905c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42941 } }
I20260619 07:02:31.687471 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:31.682846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36395 (local address 127.11.106.126:42941)
0619 07:02:31.682961 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:31.682966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:31.683147 (+ 181us) server_negotiation.cc:407] Connection header received
0619 07:02:31.683236 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:31.683238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:31.683284 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:31.683359 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:31.683590 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:31.684796 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:31.685734 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:31.686076 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:31.686199 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:31.687053 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:31.687057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:31.687060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:31.687072 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:31.687101 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:31.687119 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:31.687203 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:31.687277 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 07:02:31.687335 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 07:02:31.687677 1343 leader_election.cc:302] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54db1b3ba0d94f77aeee1b684c2b905c; no voters:
I20260619 07:02:31.688042 1343 leader_election.cc:288] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:31.688110 1353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:31.688334 1353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [term 1 LEADER]: Becoming Leader. State: Replica: 54db1b3ba0d94f77aeee1b684c2b905c, State: Running, Role: LEADER
I20260619 07:02:31.688666 1353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db1b3ba0d94f77aeee1b684c2b905c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42941 } }
I20260619 07:02:31.689435 1343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:31.689874 1353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54db1b3ba0d94f77aeee1b684c2b905c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db1b3ba0d94f77aeee1b684c2b905c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42941 } } }
I20260619 07:02:31.689990 1353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:31.690348 1354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54db1b3ba0d94f77aeee1b684c2b905c. Latest consensus state: current_term: 1 leader_uuid: "54db1b3ba0d94f77aeee1b684c2b905c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54db1b3ba0d94f77aeee1b684c2b905c" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 42941 } } }
I20260619 07:02:31.690416 1357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:31.690510 1354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:31.693679 1357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:31.699909 1357 catalog_manager.cc:1347] Generated new cluster ID: 825ff258cacb4366ab334790fd7681a4
I20260619 07:02:31.699987 1357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:31.722543 1357 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:31.723871 1357 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:31.739609 1357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54db1b3ba0d94f77aeee1b684c2b905c: Generated new TSK 0
I20260619 07:02:31.740476 1357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:31.745625 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 1275
2026-06-19T07:02:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:31 dist-test-slave-10d0 krb5kdc[1374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:31 dist-test-slave-10d0 krb5kdc[1374](info): set up 2 sockets
Jun 19 07:02:31 dist-test-slave-10d0 krb5kdc[1374](info): commencing operation
krb5kdc: starting...
W20260619 07:02:32.797482 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:32 dist-test-slave-10d0 krb5kdc[1374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:32.917807 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38429
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:46531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:38429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:33.029227 1387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:33.029492 1387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:33.029536 1387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:33.033560 1387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:33.033622 1387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:33.033641 1387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:33.033658 1387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:33.033674 1387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:33.039018 1387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:46531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:38429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:33.040030 1387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:33.041029 1387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:33.049723 1395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:33.049732 1393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:33.049793 1392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:33.050177 1387 server_base.cc:1034] running on GCE node
I20260619 07:02:33.050727 1387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:33.051826 1387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:33.053014 1387 hybrid_clock.cc:648] HybridClock initialized: now 1781852553053004 us; error 32 us; skew 500 ppm
Jun 19 07:02:33 dist-test-slave-10d0 krb5kdc[1374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852553, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:33.055691 1387 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:33.056674 1387 webserver.cc:460] Webserver started at http://127.11.106.126:34657/ using document root <none> and password file <none>
I20260619 07:02:33.057267 1387 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:33.057363 1387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:33.057606 1387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:33.059654 1387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "9d83e8059eeb4ea285bde2ca64b4ab59"
format_stamp: "Formatted at 2026-06-19 07:02:33 on dist-test-slave-10d0"
server_key: "6a84434ac02bc696a26c07e85712f436"
server_key_iv: "7f477d837c1f97795d383882dd0cf5fc"
server_key_version: "encryptionkey@0"
I20260619 07:02:33.060213 1387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "9d83e8059eeb4ea285bde2ca64b4ab59"
format_stamp: "Formatted at 2026-06-19 07:02:33 on dist-test-slave-10d0"
server_key: "6a84434ac02bc696a26c07e85712f436"
server_key_iv: "7f477d837c1f97795d383882dd0cf5fc"
server_key_version: "encryptionkey@0"
I20260619 07:02:33.063998 1387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 07:02:33.066563 1403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:33.067811 1387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:33.067965 1387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "9d83e8059eeb4ea285bde2ca64b4ab59"
format_stamp: "Formatted at 2026-06-19 07:02:33 on dist-test-slave-10d0"
server_key: "6a84434ac02bc696a26c07e85712f436"
server_key_iv: "7f477d837c1f97795d383882dd0cf5fc"
server_key_version: "encryptionkey@0"
I20260619 07:02:33.068085 1387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:33.072706 1387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:33.075518 1387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:33.075711 1387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:33.082839 1387 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:38429
I20260619 07:02:33.082861 1455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:38429 every 8 connection(s)
I20260619 07:02:33.083809 1387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:33.086529 1456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:33.092027 1456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59: Bootstrap starting.
I20260619 07:02:33.093451 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 1387
I20260619 07:02:33.093611 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:33.093885 11689 external_mini_cluster.cc:1428] Setting key 40ae6960ea01ecbc88462dc27d38de1c
I20260619 07:02:33.094964 1456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:33.095808 1456 log.cc:824] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:33.097887 1456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59: No bootstrap required, opened a new log
I20260619 07:02:33.099076 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:33.095026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56243 (local address 127.11.106.126:38429)
0619 07:02:33.095500 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:33.095514 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:33.095549 (+ 35us) server_negotiation.cc:407] Connection header received
0619 07:02:33.096274 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:33.096298 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:33.096685 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:33.097027 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:33.097805 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:33.097810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:33.097822 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:33.097833 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:33.097886 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:33.097905 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:33.098032 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:33.098195 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 07:02:33.098364 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":157,"threads_started":1}
I20260619 07:02:33.102056 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:33.101975 1456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d83e8059eeb4ea285bde2ca64b4ab59" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38429 } }
I20260619 07:02:33.102267 1456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 07:02:33.102329 1456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d83e8059eeb4ea285bde2ca64b4ab59, State: Initialized, Role: FOLLOWER
I20260619 07:02:33.102797 1456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d83e8059eeb4ea285bde2ca64b4ab59" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38429 } }
I20260619 07:02:33.102968 1456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 07:02:33.103051 1456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 07:02:33.103163 1456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:33.104373 1456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d83e8059eeb4ea285bde2ca64b4ab59" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38429 } }
I20260619 07:02:33.104838 1456 leader_election.cc:302] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d83e8059eeb4ea285bde2ca64b4ab59; no voters:
I20260619 07:02:33.105252 1456 leader_election.cc:288] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 07:02:33.105311 1466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:33.105501 1466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [term 1 LEADER]: Becoming Leader. State: Replica: 9d83e8059eeb4ea285bde2ca64b4ab59, State: Running, Role: LEADER
I20260619 07:02:33.105799 1466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d83e8059eeb4ea285bde2ca64b4ab59" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38429 } }
I20260619 07:02:33.106261 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:33.104355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42111 (local address 127.11.106.126:38429)
0619 07:02:33.104500 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:33.104506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:33.104806 (+ 300us) server_negotiation.cc:407] Connection header received
0619 07:02:33.104927 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:33.104930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:33.104986 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:33.105070 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:33.105842 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:33.105846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:33.105848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 07:02:33.105860 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:33.105887 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 07:02:33.105899 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:33.105970 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:33.106106 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 07:02:33.106144 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 07:02:33.106567 1456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:33.107501 1467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d83e8059eeb4ea285bde2ca64b4ab59. Latest consensus state: current_term: 1 leader_uuid: "9d83e8059eeb4ea285bde2ca64b4ab59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d83e8059eeb4ea285bde2ca64b4ab59" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38429 } } }
I20260619 07:02:33.107657 1467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:33.107887 1467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d83e8059eeb4ea285bde2ca64b4ab59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d83e8059eeb4ea285bde2ca64b4ab59" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 38429 } } }
I20260619 07:02:33.107966 1467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:33.108047 1475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:33.111451 1475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:33.117105 1475 catalog_manager.cc:1347] Generated new cluster ID: 61dba80080b14c3eb798f23c15456171
I20260619 07:02:33.117178 1475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:33.136343 1475 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:33.136739 1475 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:33.148057 1475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d83e8059eeb4ea285bde2ca64b4ab59: Generated new TSK 0
I20260619 07:02:33.149004 1475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 07:02:33.153931 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 1387
2026-06-19T07:02:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1409 ms)
[----------] 18 tests from AuthTokenIssuingTest (25689 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:33 dist-test-slave-10d0 krb5kdc[1487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:33 dist-test-slave-10d0 krb5kdc[1487](info): set up 2 sockets
Jun 19 07:02:33 dist-test-slave-10d0 krb5kdc[1487](info): commencing operation
krb5kdc: starting...
W20260619 07:02:34.185360 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:34.307746 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:41815
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:41815,127.11.106.125:33889,127.11.106.124:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:34.418810 1500 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 07:02:34.419127 1500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:34.419183 1500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:34.419253 1500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:34.423273 1500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:34.423357 1500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:34.423411 1500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:34.423468 1500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:34.423506 1500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:34.429013 1500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:41815,127.11.106.125:33889,127.11.106.124:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:41815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:34.430080 1500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:34.431143 1500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:34.439754 1506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:34.439760 1505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:34.439765 1508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:34.440057 1500 server_base.cc:1034] running on GCE node
I20260619 07:02:34.440663 1500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:34.441839 1500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:34.443030 1500 hybrid_clock.cc:648] HybridClock initialized: now 1781852554443002 us; error 48 us; skew 500 ppm
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:34.445931 1500 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:34.446765 1500 webserver.cc:460] Webserver started at http://127.11.106.126:37605/ using document root <none> and password file <none>
I20260619 07:02:34.447306 1500 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:34.447398 1500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:34.447628 1500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:34.449733 1500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "62dd7b73d57a4a23bc4f141b7424f5ba"
format_stamp: "Formatted at 2026-06-19 07:02:34 on dist-test-slave-10d0"
server_key: "9d3fcbbac6454dbf6ea0fe4b0834306f"
server_key_iv: "02b54cc245c1f42353f4f96f21faeed6"
server_key_version: "encryptionkey@0"
I20260619 07:02:34.450294 1500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "62dd7b73d57a4a23bc4f141b7424f5ba"
format_stamp: "Formatted at 2026-06-19 07:02:34 on dist-test-slave-10d0"
server_key: "9d3fcbbac6454dbf6ea0fe4b0834306f"
server_key_iv: "02b54cc245c1f42353f4f96f21faeed6"
server_key_version: "encryptionkey@0"
I20260619 07:02:34.453954 1500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 07:02:34.456368 1515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:34.457662 1500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:34.457805 1500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "62dd7b73d57a4a23bc4f141b7424f5ba"
format_stamp: "Formatted at 2026-06-19 07:02:34 on dist-test-slave-10d0"
server_key: "9d3fcbbac6454dbf6ea0fe4b0834306f"
server_key_iv: "02b54cc245c1f42353f4f96f21faeed6"
server_key_version: "encryptionkey@0"
I20260619 07:02:34.457906 1500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:34.468700 1500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:34.471664 1500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:34.471863 1500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:34.478645 1500 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:41815
I20260619 07:02:34.478924 1567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:41815 every 8 connection(s)
I20260619 07:02:34.479594 1500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:34.482425 1568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:34.483566 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 1500
I20260619 07:02:34.483712 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:34.484012 11689 external_mini_cluster.cc:1428] Setting key b715e190ec6f6795448ad461221e1a45
I20260619 07:02:34.487141 1568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.106.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.125@KRBTEST.COM" created.
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:34.503257 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.489579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:41815 (local address 127.0.0.1:43914)
0619 07:02:34.490147 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.490184 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:34.490995 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:34.491677 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:34.491682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:34.492120 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:34.492355 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.492369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.493590 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.493600 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.494501 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.494511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.495381 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.495385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.495546 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.496149 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:34.496166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:34.497874 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:34.500292 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.500301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.500310 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.500584 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.500988 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.500990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.500991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.501104 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.501875 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:34.501880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:34.502004 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 07:02:34.502144 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 07:02:34.502408 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":39,"spinlock_wait_cycles":2304,"thread_start_us":162,"threads_started":1}
I20260619 07:02:34.503261 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.489571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.11.106.126:41815)
0619 07:02:34.489986 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.490006 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.490293 (+ 287us) server_negotiation.cc:407] Connection header received
0619 07:02:34.491305 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.491310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.491486 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.491681 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.492505 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.493457 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.494661 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.495201 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.495398 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.498019 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.498025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.498039 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.498068 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.500175 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.500707 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.500716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.500719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.500863 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.501203 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.501207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.501209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.501756 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.501909 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.502265 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.502484 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":261,"spinlock_wait_cycles":1920,"thread_start_us":125,"threads_started":1}
I20260619 07:02:34.505748 1568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:34.506435 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.506202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:33889 (local address 127.0.0.1:46174)
0619 07:02:34.506314 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.506380 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 07:02:34.506633 1518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
I20260619 07:02:34.507604 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.507417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:33889 (local address 127.0.0.1:46190)
0619 07:02:34.507511 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.507553 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 07:02:34.507750 1568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.125:33889: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab.
Entry for principal kudu/127.11.106.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab.
I20260619 07:02:34.511848 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.125:33889
--webserver_interface=127.11.106.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab
--principal=kudu/127.11.106.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:41815,127.11.106.125:33889,127.11.106.124:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:02:34.557014 1568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } attempt: 1
I20260619 07:02:34.557897 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.557600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:33889 (local address 127.0.0.1:46194)
0619 07:02:34.557754 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.557819 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 07:02:34.558712 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.558491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:33889 (local address 127.0.0.1:46198)
0619 07:02:34.558601 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.558649 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 07:02:34.558887 1568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.125:33889: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
W20260619 07:02:34.620139 1575 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 07:02:34.620414 1575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:34.620440 1575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:34.620581 1575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:34.624490 1575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:34.624572 1575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:34.624593 1575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:34.624609 1575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:34.624624 1575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:34.627043 1568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } attempt: 2
I20260619 07:02:34.627878 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.627583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:33889 (local address 127.0.0.1:46210)
0619 07:02:34.627736 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.627801 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 07:02:34.628734 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.628455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:33889 (local address 127.0.0.1:46216)
0619 07:02:34.628629 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.628678 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 07:02:34.628883 1568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.125:33889: Network error: Client connection negotiation failed: client connection to 127.11.106.125:33889: connect: Connection refused (error 111)
I20260619 07:02:34.630824 1575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:41815,127.11.106.125:33889,127.11.106.124:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab
--principal=kudu/127.11.106.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.125:33889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.106.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:34.631901 1575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:34.632989 1575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:34.641315 1581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:34.641353 1580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:34.641666 1575 server_base.cc:1034] running on GCE node
W20260619 07:02:34.641369 1583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:34.642253 1575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:34.643412 1575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:34.644616 1575 hybrid_clock.cc:648] HybridClock initialized: now 1781852554644599 us; error 30 us; skew 500 ppm
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:34.647353 1575 init.cc:377] Logged in from keytab as kudu/127.11.106.125@KRBTEST.COM (short username kudu)
I20260619 07:02:34.648336 1575 webserver.cc:460] Webserver started at http://127.11.106.125:41795/ using document root <none> and password file <none>
I20260619 07:02:34.648991 1575 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:34.649055 1575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:34.649302 1575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:34.651377 1575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/data/instance:
uuid: "ee3363cb334a40758ec695721c9a6016"
format_stamp: "Formatted at 2026-06-19 07:02:34 on dist-test-slave-10d0"
server_key: "a5552fc177ed09ad4334e4ba07f3f76d"
server_key_iv: "0ed2018a2fb8067015e6fc14765e11dd"
server_key_version: "encryptionkey@0"
I20260619 07:02:34.651939 1575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/wal/instance:
uuid: "ee3363cb334a40758ec695721c9a6016"
format_stamp: "Formatted at 2026-06-19 07:02:34 on dist-test-slave-10d0"
server_key: "a5552fc177ed09ad4334e4ba07f3f76d"
server_key_iv: "0ed2018a2fb8067015e6fc14765e11dd"
server_key_version: "encryptionkey@0"
I20260619 07:02:34.655726 1575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 07:02:34.658280 1590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:34.659533 1575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 07:02:34.659660 1575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/wal
uuid: "ee3363cb334a40758ec695721c9a6016"
format_stamp: "Formatted at 2026-06-19 07:02:34 on dist-test-slave-10d0"
server_key: "a5552fc177ed09ad4334e4ba07f3f76d"
server_key_iv: "0ed2018a2fb8067015e6fc14765e11dd"
server_key_version: "encryptionkey@0"
I20260619 07:02:34.659768 1575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:34.700084 1575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:34.703089 1575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:34.703299 1575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:34.710579 1575 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.125:33889
I20260619 07:02:34.710872 1643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.125:33889 every 8 connection(s)
I20260619 07:02:34.711602 1575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
I20260619 07:02:34.714325 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:34.717943 1644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:34.718214 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 1575
I20260619 07:02:34.718353 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-1/wal/instance
I20260619 07:02:34.718617 11689 external_mini_cluster.cc:1428] Setting key 8f7f05eb5dc72387691ece902dd9dd47
I20260619 07:02:34.720110 1568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } attempt: 3
WARNING: no policy specified for kudu/127.11.106.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.124@KRBTEST.COM" created.
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
I20260619 07:02:34.735958 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.720521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:33889 (local address 127.0.0.1:46232)
0619 07:02:34.720750 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.720768 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:34.720864 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:34.722769 (+ 1905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:34.722772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:34.723005 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:34.723173 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.723181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.725246 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.725250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.726184 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.726195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.727098 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.727102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.727218 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.727769 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:34.727789 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:34.731444 (+ 3655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:34.733926 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.733929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.733932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.734160 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.734545 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.734548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.734550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.734640 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.735330 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:34.735333 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:34.735422 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 07:02:34.735547 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 07:02:34.735717 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260619 07:02:34.735965 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.719901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.11.106.126:41815)
0619 07:02:34.720069 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.720075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.721268 (+ 1193us) server_negotiation.cc:407] Connection header received
0619 07:02:34.722493 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.722497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.722558 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.722648 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.723827 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.724343 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.725672 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.726040 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.726116 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.731236 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.731243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.731249 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.731283 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.733636 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.734276 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.734283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.734288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.734384 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.734781 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.734784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.734786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.734999 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.735105 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.735600 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.735806 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":83}
I20260619 07:02:34.737036 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.720635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.11.106.125:33889)
0619 07:02:34.721132 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.721153 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.721196 (+ 43us) server_negotiation.cc:407] Connection header received
0619 07:02:34.722285 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.722312 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.722566 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.723338 (+ 772us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.723571 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.725107 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.726312 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.726963 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.727088 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.731637 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.731645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.731661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.731694 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.733806 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.734288 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.734293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.734294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.734449 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.734741 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.734747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.734750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.735226 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.735372 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.736659 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.736772 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":364,"thread_start_us":116,"threads_started":1}
I20260619 07:02:34.737044 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.720319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:41815 (local address 127.0.0.1:43922)
0619 07:02:34.721116 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.721163 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:34.722293 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:34.722700 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:34.722709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:34.723174 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:34.723690 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.723701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.724454 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.724459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.725512 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.725524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.726313 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.726317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.726494 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.727078 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:34.727099 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:34.731055 (+ 3956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:34.733774 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.733784 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.733794 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.734142 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.734511 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.734516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.734521 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.734691 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.735130 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:34.735139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:34.735327 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 07:02:34.735530 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 07:02:34.735916 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":107,"spinlock_wait_cycles":1792,"thread_start_us":168,"threads_started":1}
I20260619 07:02:34.738490 1644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:34.739450 1568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:34.740119 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.739883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55232)
0619 07:02:34.739996 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.740049 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 07:02:34.740932 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.740684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55236)
0619 07:02:34.740839 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.740883 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 07:02:34.741086 1568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36245: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab.
Entry for principal kudu/127.11.106.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab.
I20260619 07:02:34.746934 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.738998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:33889 (local address 127.0.0.1:46244)
0619 07:02:34.739148 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.739166 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:34.739249 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:34.739666 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:34.739669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:34.739851 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:34.739958 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.739964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.740982 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.740988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.741667 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.741677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.742189 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.742193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.742317 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.742741 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:34.742753 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:34.743885 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:34.745587 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.745590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.745591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.745772 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.746027 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.746029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.746031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.746113 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.746512 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:34.746514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:34.746594 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 07:02:34.746676 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 07:02:34.746812 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":22}
I20260619 07:02:34.747109 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.739299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.11.106.125:33889)
0619 07:02:34.739419 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.739423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.739436 (+ 13us) server_negotiation.cc:407] Connection header received
0619 07:02:34.739471 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.739475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.739516 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.739597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.740082 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.740873 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.741793 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.742077 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.742191 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.744027 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.744031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.744033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.744055 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.745452 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.745856 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.745859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.745860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.745932 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.746213 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.746215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.746216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.746374 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.746489 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.746760 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.746890 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":50}
I20260619 07:02:34.747743 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.124:36245
--webserver_interface=127.11.106.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab
--principal=kudu/127.11.106.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:40261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:41815,127.11.106.125:33889,127.11.106.124:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:02:34.747956 1644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:34.748711 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.748422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55242)
0619 07:02:34.748529 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.748662 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 07:02:34.749397 1594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
I20260619 07:02:34.750316 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.750097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55248)
0619 07:02:34.750225 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.750269 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 07:02:34.750578 1644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36245: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
I20260619 07:02:34.772262 1568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } attempt: 1
I20260619 07:02:34.773083 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.772805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55264)
0619 07:02:34.772950 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.773008 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 07:02:34.773830 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.773619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55270)
0619 07:02:34.773725 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.773775 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 07:02:34.773958 1568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36245: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
I20260619 07:02:34.799937 1644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } attempt: 1
I20260619 07:02:34.800868 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.800593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55272)
0619 07:02:34.800737 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.800796 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 07:02:34.801559 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.801354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55274)
0619 07:02:34.801459 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.801508 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 07:02:34.801695 1644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36245: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
I20260619 07:02:34.849112 1568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } attempt: 2
I20260619 07:02:34.849876 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.849594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55288)
0619 07:02:34.849737 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.849802 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 07:02:34.850606 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.850389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55290)
0619 07:02:34.850497 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.850547 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 07:02:34.850718 1568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36245: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
W20260619 07:02:34.859117 1650 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 07:02:34.859419 1650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:34.859488 1650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:34.859577 1650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:34.863724 1650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:34.863793 1650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:34.863839 1650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:34.863880 1650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:34.863934 1650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:34.869763 1650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:40261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:41815,127.11.106.125:33889,127.11.106.124:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab
--principal=kudu/127.11.106.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.124:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.106.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:34.870741 1644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } attempt: 2
I20260619 07:02:34.871052 1650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:34.871531 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.871258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55292)
0619 07:02:34.871391 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.871453 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 07:02:34.872190 1650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 07:02:34.872205 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.871988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55306)
0619 07:02:34.872100 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.872149 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 07:02:34.872394 1644 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:36245: Network error: Client connection negotiation failed: client connection to 127.11.106.124:36245: connect: Connection refused (error 111)
W20260619 07:02:34.880630 1659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:34.880795 1656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:34.880816 1657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:34.880780 1650 server_base.cc:1034] running on GCE node
I20260619 07:02:34.881520 1650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:34.882563 1650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:34.883735 1650 hybrid_clock.cc:648] HybridClock initialized: now 1781852554883701 us; error 44 us; skew 500 ppm
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:34.886448 1650 init.cc:377] Logged in from keytab as kudu/127.11.106.124@KRBTEST.COM (short username kudu)
I20260619 07:02:34.887351 1650 webserver.cc:460] Webserver started at http://127.11.106.124:43319/ using document root <none> and password file <none>
I20260619 07:02:34.887898 1650 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:34.887960 1650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:34.888198 1650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:34.890252 1650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/data/instance:
uuid: "428f95c923934e9496233d971456f8d5"
format_stamp: "Formatted at 2026-06-19 07:02:34 on dist-test-slave-10d0"
server_key: "60fdbfbc523813ba41abc12e61b77141"
server_key_iv: "c44efcba5039949fcff20b1a74685a22"
server_key_version: "encryptionkey@0"
I20260619 07:02:34.890815 1650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/wal/instance:
uuid: "428f95c923934e9496233d971456f8d5"
format_stamp: "Formatted at 2026-06-19 07:02:34 on dist-test-slave-10d0"
server_key: "60fdbfbc523813ba41abc12e61b77141"
server_key_iv: "c44efcba5039949fcff20b1a74685a22"
server_key_version: "encryptionkey@0"
I20260619 07:02:34.894524 1650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 07:02:34.896904 1666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:34.898100 1650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 07:02:34.898227 1650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/wal
uuid: "428f95c923934e9496233d971456f8d5"
format_stamp: "Formatted at 2026-06-19 07:02:34 on dist-test-slave-10d0"
server_key: "60fdbfbc523813ba41abc12e61b77141"
server_key_iv: "c44efcba5039949fcff20b1a74685a22"
server_key_version: "encryptionkey@0"
I20260619 07:02:34.898329 1650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:34.910308 1650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:34.912961 1650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:34.913151 1650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:34.919927 1650 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.124:36245
I20260619 07:02:34.920200 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.124:36245 every 8 connection(s)
I20260619 07:02:34.920899 1650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
I20260619 07:02:34.923596 1720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:34.924450 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 1650
I20260619 07:02:34.924623 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781852273483449-11689-0/minicluster-data/master-2/wal/instance
I20260619 07:02:34.924922 11689 external_mini_cluster.cc:1428] Setting key 4ad79596781239906b81eb044b9d5b6b
I20260619 07:02:34.927719 1720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:34.939133 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.929584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.11.106.126:41815)
0619 07:02:34.929734 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.929738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.930025 (+ 287us) server_negotiation.cc:407] Connection header received
0619 07:02:34.930160 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.930163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.930211 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.930302 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.930914 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.931392 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.932390 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.932700 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.932813 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.935512 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.935514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.935516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.935538 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.937010 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.937568 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.937579 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.937581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.937708 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.938046 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.938049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.938050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.938309 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.938423 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.938814 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.938955 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 07:02:34.939941 1724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.929513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:41815 (local address 127.0.0.1:43934)
0619 07:02:34.929892 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.929938 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:34.930054 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:34.930353 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:34.930357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:34.930663 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:34.930780 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.930787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.931529 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.931534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.932238 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.932248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.932847 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.932851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.932952 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.933420 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:34.933435 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:34.935354 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:34.937120 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.937128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.937135 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.937461 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.937819 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.937822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.937824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.937956 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.938422 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:34.938426 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:34.938543 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 07:02:34.938740 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 07:02:34.939061 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":163,"threads_started":1}
I20260619 07:02:34.939944 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.926317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.11.106.124:36245)
0619 07:02:34.926759 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.926778 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.926820 (+ 42us) server_negotiation.cc:407] Connection header received
0619 07:02:34.927626 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.927648 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.928055 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.928413 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.929547 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.930793 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.931757 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.932209 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.932324 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.934958 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.934965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.934978 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.935003 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.937127 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.937575 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.937581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.937583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.937724 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.938024 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.938028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.938031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.938541 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.938722 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.938956 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.939331 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":293,"thread_start_us":112,"threads_started":1}
I20260619 07:02:34.941171 1720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:34.949873 1568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } attempt: 3
I20260619 07:02:34.950441 1724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.941673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:33889 (local address 127.0.0.1:46250)
0619 07:02:34.941769 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.941782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:34.941852 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:34.942098 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:34.942100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:34.942284 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:34.942395 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.942404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.943432 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.943435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.944251 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.944259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.944840 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.944842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.944914 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.945369 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:34.945381 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:34.946806 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:34.948722 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.948724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.948726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.948970 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.949292 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.949294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.949296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.949370 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.949817 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:34.949819 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:34.949955 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 07:02:34.950145 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 07:02:34.950278 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 07:02:34.950614 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.941762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.11.106.125:33889)
0619 07:02:34.941883 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.941889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.941904 (+ 15us) server_negotiation.cc:407] Connection header received
0619 07:02:34.941949 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.941957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.941993 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.942080 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.942524 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.943327 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.944352 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.944736 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.944841 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.946957 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.946961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.946964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.946991 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.948615 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.949088 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.949094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.949098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.949190 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.949466 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.949469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.949471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.949718 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.949822 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.950205 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.950315 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 07:02:34.951545 1720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:34.951709 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.942343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.11.106.126:41815)
0619 07:02:34.942471 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.942477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.942493 (+ 16us) server_negotiation.cc:407] Connection header received
0619 07:02:34.942574 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.942576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.942607 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.942662 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.943327 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.944074 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.945108 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.945328 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.945435 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.948103 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.948110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.948114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.948142 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.949825 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.950259 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.950262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.950264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.950347 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.950661 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.950666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.950669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.950871 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.951094 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.951433 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.951546 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
I20260619 07:02:34.959112 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.950278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55332)
0619 07:02:34.950546 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.950560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:34.950631 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:34.951108 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:34.951114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:34.951280 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:34.951383 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.951390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.952329 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.952334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.953102 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.953115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.953754 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.953760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.953846 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.954285 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:34.954299 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:34.955686 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:34.957595 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.957598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.957600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.957802 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.958078 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.958084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.958089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.958182 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.958582 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:34.958588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:34.958701 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 07:02:34.958817 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 07:02:34.958945 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260619 07:02:34.959134 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.950528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.11.106.124:36245)
0619 07:02:34.950880 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.950885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.950898 (+ 13us) server_negotiation.cc:407] Connection header received
0619 07:02:34.950933 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.950936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.950999 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.951081 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.951513 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.952230 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.953209 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.953598 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.954039 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.955820 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.955824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.955826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.955853 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.957466 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.957902 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.957907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.957909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.957986 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.958310 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.958312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.958313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.958464 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.958568 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.958881 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.958982 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":280}
I20260619 07:02:34.962772 1724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.951961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55342)
0619 07:02:34.952089 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.952105 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:34.952188 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:34.953255 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:34.953258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:34.953384 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:34.953437 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.953441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.954290 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.954294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.955088 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.955099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.955730 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.955735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.956353 (+ 618us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.956931 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:34.956946 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:34.958954 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:34.961189 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.961192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.961194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.961470 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.961732 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.961737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.961740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.961819 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.962281 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:34.962284 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:34.962364 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 07:02:34.962507 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 07:02:34.962623 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":516}
I20260619 07:02:34.962772 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.952916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.11.106.125:33889)
0619 07:02:34.953019 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.953023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.953035 (+ 12us) server_negotiation.cc:407] Connection header received
0619 07:02:34.953132 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.953134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.953176 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.953244 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.954282 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.954831 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.955785 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.956074 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.956188 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.959180 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.959186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.959191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.959217 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.961054 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.961502 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.961508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.961513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.961600 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.961885 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.961890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.961895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.962120 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.962319 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.962497 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.962626 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 07:02:34.963248 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.952637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.11.106.124:36245)
0619 07:02:34.952973 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.952984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.952997 (+ 13us) server_negotiation.cc:407] Connection header received
0619 07:02:34.953029 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.953031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.953068 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.953156 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.953550 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.954180 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.955210 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.955605 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.955717 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.959098 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.959104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.959109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.959137 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.961090 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.961571 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.961582 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.961583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.961659 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.961933 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.961938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.961943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.962169 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.962248 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.962960 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.963069 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260619 07:02:34.963320 1568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba: Bootstrap starting.
I20260619 07:02:34.963527 1644 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } attempt: 3
I20260619 07:02:34.966337 1568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:34.966408 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:34.967223 1568 log.cc:824] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba: Log is configured to *not* fsync() on all Append() calls
Jun 19 07:02:34 dist-test-slave-10d0 krb5kdc[1487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852554, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
I20260619 07:02:34.967621 1542 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43950
I20260619 07:02:34.970285 1568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba: No bootstrap required, opened a new log
I20260619 07:02:34.970966 1720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5: Bootstrap starting.
I20260619 07:02:34.972738 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.963935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:36245 (local address 127.0.0.1:55356)
0619 07:02:34.964035 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:34.964051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:34.964158 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:34.964429 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:34.964432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:34.964651 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:34.964759 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.964767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.965731 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.965734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.966495 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:34.966508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.967130 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.967134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:34.967213 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.967685 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:34.967705 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:34.969258 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:34.971188 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.971191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.971193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.971422 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.971743 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:34.971746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:34.971748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:34.971832 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:34.972265 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:34.972270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:34.972363 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 07:02:34.972442 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 07:02:34.972593 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 07:02:34.973323 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.964053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.11.106.124:36245)
0619 07:02:34.964160 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.964165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.964194 (+ 29us) server_negotiation.cc:407] Connection header received
0619 07:02:34.964234 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.964239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.964277 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.964386 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.964899 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.965600 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.966658 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.966978 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.967431 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:34.969401 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:34.969404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:34.969406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:34.969428 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:34.971060 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.971545 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.971548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.971550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.971641 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:34.971942 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:34.971944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:34.971946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:34.972147 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:34.972991 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:34.973063 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 07:02:34.973166 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 07:02:34.973280 1568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } }
I20260619 07:02:34.973510 1568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dd7b73d57a4a23bc4f141b7424f5ba, State: Initialized, Role: FOLLOWER
I20260619 07:02:34.973728 1720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:34.973841 1568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } }
I20260619 07:02:34.974668 1720 log.cc:824] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:34.975168 1568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:34.975252 1542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 07:02:34.975342 1542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:34.975881 1644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016: Bootstrap starting.
I20260619 07:02:34.976649 1542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } }
I20260619 07:02:34.977466 1727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:34.977592 1727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:34.977608 1542 leader_election.cc:288] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [CANDIDATE]: Term 1 election: Requested vote from peers ee3363cb334a40758ec695721c9a6016 (127.11.106.125:33889), 428f95c923934e9496233d971456f8d5 (127.11.106.124:36245)
I20260619 07:02:34.978377 1644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:34.979174 1644 log.cc:824] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016: Log is configured to *not* fsync() on all Append() calls
W20260619 07:02:34.980458 1739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:34.980533 1739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:34.981869 1720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5: No bootstrap required, opened a new log
I20260619 07:02:34.984058 1644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016: No bootstrap required, opened a new log
I20260619 07:02:34.982719 1500 master_runner.cc:186] Error getting master registration for 127.11.106.125:33889: OK, instance_id { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" instance_seqno: 1781852554708489 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 07:02:34.985967 1500 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 07:02:34.985807 1720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } }
I20260619 07:02:34.986040 1720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428f95c923934e9496233d971456f8d5, State: Initialized, Role: FOLLOWER
I20260619 07:02:34.986354 1720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } }
I20260619 07:02:34.987607 1720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:34.987680 1693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "428f95c923934e9496233d971456f8d5"
I20260619 07:02:34.987911 1693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:34.988026 1644 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } }
I20260619 07:02:34.988231 1644 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3363cb334a40758ec695721c9a6016, State: Initialized, Role: FOLLOWER
I20260619 07:02:34.988512 1644 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } }
I20260619 07:02:34.989207 1693 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62dd7b73d57a4a23bc4f141b7424f5ba in term 1.
I20260619 07:02:34.989748 1519 leader_election.cc:302] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428f95c923934e9496233d971456f8d5, 62dd7b73d57a4a23bc4f141b7424f5ba; no voters:
I20260619 07:02:34.990026 1727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:34.990049 1644 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:34.990229 1727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [term 1 LEADER]: Becoming Leader. State: Replica: 62dd7b73d57a4a23bc4f141b7424f5ba, State: Running, Role: LEADER
I20260619 07:02:34.990528 1727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } }
I20260619 07:02:34.992354 1727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62dd7b73d57a4a23bc4f141b7424f5ba. Latest consensus state: current_term: 1 leader_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:34.992499 1727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:34.993323 1617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ee3363cb334a40758ec695721c9a6016"
I20260619 07:02:34.993532 1617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:34.994817 1617 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62dd7b73d57a4a23bc4f141b7424f5ba in term 1.
I20260619 07:02:35.000718 1749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:35.000859 1749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:35.000941 1775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:35.001992 1775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 07:02:35.004487 1761 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:35.004588 1761 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:35.004858 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.981386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.11.106.124:36245)
0619 07:02:34.990233 (+ 8847us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.990237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.995859 (+ 5622us) server_negotiation.cc:407] Connection header received
0619 07:02:34.995977 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.995983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.996039 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.996092 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.997122 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.997633 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.998634 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.998933 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.999011 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:35.001011 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:35.001017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:35.001021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:35.001049 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:35.002910 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:35.003458 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:35.003462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:35.003463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:35.003547 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:35.003874 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:35.003877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:35.003879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:35.004088 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:35.004216 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:35.004592 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 07:02:35.004711 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8762}
I20260619 07:02:35.008611 1693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [term 1 FOLLOWER]: Refusing update from remote peer 62dd7b73d57a4a23bc4f141b7424f5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:35.009384 1727 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:35.009876 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.980758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.11.106.126:41815)
0619 07:02:34.980864 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.980868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.981561 (+ 693us) server_negotiation.cc:407] Connection header received
0619 07:02:34.981948 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.981954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.982004 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.982059 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:34.984356 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:34.985101 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:34.999847 (+ 14746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:35.000186 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:35.000274 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:35.005979 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:35.005984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:35.005986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:35.006014 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:35.007926 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:35.008477 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:35.008481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:35.008482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:35.008606 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:35.008908 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:35.008911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:35.008913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:35.009126 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:35.009228 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:35.009591 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 07:02:35.009720 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 07:02:35.009997 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:34.981934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.11.106.125:33889)
0619 07:02:34.982040 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:34.982046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:34.999599 (+ 17553us) server_negotiation.cc:407] Connection header received
0619 07:02:34.999734 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:34.999737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:34.999789 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:34.999862 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.000795 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:35.001468 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:35.002526 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:35.002843 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:35.002915 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:35.006019 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:35.006023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:35.006025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:35.006051 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:35.008034 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:35.008579 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:35.008582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:35.008584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:35.008677 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:35.008991 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:35.008996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:35.008998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:35.009243 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:35.009353 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:35.009700 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 07:02:35.009826 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 07:02:35.014021 1749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62dd7b73d57a4a23bc4f141b7424f5ba. Latest consensus state: current_term: 1 leader_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:35.014140 1749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 07:02:35.017795 1773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:35.017897 1773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:35.019192 1617 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [term 1 FOLLOWER]: Refusing update from remote peer 62dd7b73d57a4a23bc4f141b7424f5ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:35.019760 1762 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:35.020377 1754 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:35.020499 1754 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:35.024611 1754 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62dd7b73d57a4a23bc4f141b7424f5ba. Latest consensus state: current_term: 1 leader_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:35.024700 1754 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:35.026715 1777 mvcc.cc:205] Tried to move back new op lower bound from 7298468065308000256 to 7298468065244868608. Current Snapshot: MvccSnapshot[applied={T|T < 7298468065308000256}]
I20260619 07:02:35.027164 1727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:35.027285 1727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:35.028438 1749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:35.028578 1749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:35.028993 1775 catalog_manager.cc:1347] Generated new cluster ID: 5e8a96ef3dae4e61b0a7bd13b4225b58
I20260619 07:02:35.029057 1775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:35.033018 1778 mvcc.cc:205] Tried to move back new op lower bound from 7298468065308000256 to 7298468065244868608. Current Snapshot: MvccSnapshot[applied={T|T < 7298468065308000256 or (T in {7298468065308000256})}]
I20260619 07:02:35.042922 1727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:35.043066 1727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:35.045269 1754 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62dd7b73d57a4a23bc4f141b7424f5ba" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 41815 } } peers { permanent_uuid: "ee3363cb334a40758ec695721c9a6016" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 33889 } } peers { permanent_uuid: "428f95c923934e9496233d971456f8d5" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 36245 } } }
I20260619 07:02:35.045395 1754 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:35.046468 1781 mvcc.cc:205] Tried to move back new op lower bound from 7298468065308000256 to 7298468065244868608. Current Snapshot: MvccSnapshot[applied={T|T < 7298468065308000256}]
I20260619 07:02:35.047487 1775 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:35.048830 1775 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:35.057536 1775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62dd7b73d57a4a23bc4f141b7424f5ba: Generated new TSK 0
I20260619 07:02:35.058386 1775 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 07:02:35.082515 1571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.079238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.11.106.126:41815)
0619 07:02:35.079366 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.079371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.081201 (+ 1830us) server_negotiation.cc:407] Connection header received
0619 07:02:35.081624 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.081629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.081685 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.081739 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.082375 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43970: BlockingRecv error: recv got EOF from 127.0.0.1:43970 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 07:02:35.083819 1725 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.079980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.11.106.124:36245)
0619 07:02:35.080091 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.080095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.081523 (+ 1428us) server_negotiation.cc:407] Connection header received
0619 07:02:35.083075 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.083078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.083126 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.083176 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.083702 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55374: BlockingRecv error: recv got EOF from 127.0.0.1:55374 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 07:02:35.085014 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.080637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.11.106.125:33889)
0619 07:02:35.080767 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.080772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.082894 (+ 2122us) server_negotiation.cc:407] Connection header received
0619 07:02:35.084166 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.084169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.084217 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.084267 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.084872 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46280: BlockingRecv error: recv got EOF from 127.0.0.1:46280 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 07:02:35.089749 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.088206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.11.106.125:33889)
0619 07:02:35.088311 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.088315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.088816 (+ 501us) server_negotiation.cc:407] Connection header received
0619 07:02:35.089028 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.089032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.089107 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.089159 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.089656 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46282: BlockingRecv error: recv got EOF from 127.0.0.1:46282 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 07:02:35.094602 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.093179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.11.106.125:33889)
0619 07:02:35.093275 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.093279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.093774 (+ 495us) server_negotiation.cc:407] Connection header received
0619 07:02:35.093951 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.093954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.094000 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.094048 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.094499 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46286: BlockingRecv error: recv got EOF from 127.0.0.1:46286 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 07:02:35.099836 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.098434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.11.106.125:33889)
0619 07:02:35.098531 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.098535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.099016 (+ 481us) server_negotiation.cc:407] Connection header received
0619 07:02:35.099190 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.099193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.099239 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.099287 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.099746 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46300: BlockingRecv error: recv got EOF from 127.0.0.1:46300 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 07:02:35.107146 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.105695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.11.106.125:33889)
0619 07:02:35.105818 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.105821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.106295 (+ 474us) server_negotiation.cc:407] Connection header received
0619 07:02:35.106466 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.106469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.106515 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.106561 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.107059 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46302: BlockingRecv error: recv got EOF from 127.0.0.1:46302 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 07:02:35.118613 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.117075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.11.106.125:33889)
0619 07:02:35.117174 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.117178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.117688 (+ 510us) server_negotiation.cc:407] Connection header received
0619 07:02:35.117877 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.117880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.117930 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.117983 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.118503 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46304: BlockingRecv error: recv got EOF from 127.0.0.1:46304 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 07:02:35.139035 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.137044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.11.106.125:33889)
0619 07:02:35.137170 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.137175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.137821 (+ 646us) server_negotiation.cc:407] Connection header received
0619 07:02:35.138084 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.138088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.138174 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.138243 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.138914 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46316: BlockingRecv error: recv got EOF from 127.0.0.1:46316 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 07:02:35.175246 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.173514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.11.106.125:33889)
0619 07:02:35.173650 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.173655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.174243 (+ 588us) server_negotiation.cc:407] Connection header received
0619 07:02:35.174448 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.174454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.174514 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.174568 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.175122 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46324: BlockingRecv error: recv got EOF from 127.0.0.1:46324 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 07:02:35.242873 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.241814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.11.106.125:33889)
0619 07:02:35.241949 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.241953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.242077 (+ 124us) server_negotiation.cc:407] Connection header received
0619 07:02:35.242210 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.242213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.242258 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.242344 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.242735 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46336: BlockingRecv error: recv got EOF from 127.0.0.1:46336 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 07:02:35.374691 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.373239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.11.106.125:33889)
0619 07:02:35.373405 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.373411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.373629 (+ 218us) server_negotiation.cc:407] Connection header received
0619 07:02:35.373821 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.373825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.373900 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.374001 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.374527 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46344: BlockingRecv error: recv got EOF from 127.0.0.1:46344 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 07:02:35.633993 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:35.632992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.11.106.125:33889)
0619 07:02:35.633154 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:35.633158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:35.633235 (+ 77us) server_negotiation.cc:407] Connection header received
0619 07:02:35.633357 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:35.633359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:35.633409 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:35.633493 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:35.633893 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46348: BlockingRecv error: recv got EOF from 127.0.0.1:46348 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 07:02:36.005928 1761 catalog_manager.cc:1259] Loaded cluster ID: 5e8a96ef3dae4e61b0a7bd13b4225b58
I20260619 07:02:36.006013 1761 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5: loading cluster ID for follower catalog manager: success
I20260619 07:02:36.007668 1761 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5: acquiring CA information for follower catalog manager: success
I20260619 07:02:36.008826 1761 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 428f95c923934e9496233d971456f8d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 07:02:36.019238 1773 catalog_manager.cc:1259] Loaded cluster ID: 5e8a96ef3dae4e61b0a7bd13b4225b58
I20260619 07:02:36.019369 1773 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016: loading cluster ID for follower catalog manager: success
I20260619 07:02:36.021121 1773 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016: acquiring CA information for follower catalog manager: success
I20260619 07:02:36.022307 1773 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee3363cb334a40758ec695721c9a6016: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 07:02:36.152395 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:36.148375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.11.106.125:33889)
0619 07:02:36.148678 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:36.148684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:36.148702 (+ 18us) server_negotiation.cc:407] Connection header received
0619 07:02:36.148898 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:36.148901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:36.149023 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:36.149152 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:36.149429 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.149957 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.150844 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.151084 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.151198 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:36.151528 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:36.151968 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:36.152114 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:36.152179 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 07:02:36.152236 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260619 07:02:36.159175 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:36.155572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.11.106.124:36245)
0619 07:02:36.155839 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:36.155842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:36.155856 (+ 14us) server_negotiation.cc:407] Connection header received
0619 07:02:36.155967 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:36.155970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:36.156097 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:36.156205 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:36.156392 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.157005 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.157779 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.158003 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.158109 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:36.158344 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:36.158762 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:36.158854 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:36.158912 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 07:02:36.158961 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260619 07:02:36.166421 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:36.162275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.11.106.125:33889)
0619 07:02:36.162418 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:36.162422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:36.162708 (+ 286us) server_negotiation.cc:407] Connection header received
0619 07:02:36.162930 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:36.162933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:36.162984 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:36.163052 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:36.163482 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.164025 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.165266 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.165547 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.165666 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:36.165936 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:36.166016 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:36.166099 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:36.166246 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 07:02:36.166300 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 07:02:36.166644 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:36.162615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.11.106.124:36245)
0619 07:02:36.162727 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:36.162733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:36.162851 (+ 118us) server_negotiation.cc:407] Connection header received
0619 07:02:36.162926 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:36.162929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:36.162978 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:36.163051 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:36.163335 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.164121 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.165300 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.165633 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.165735 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:36.166136 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:36.166216 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:36.166259 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:36.166456 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 07:02:36.166506 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 07:02:36.167398 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:36.162180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.11.106.126:41815)
0619 07:02:36.162451 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:36.162458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:36.162784 (+ 326us) server_negotiation.cc:407] Connection header received
0619 07:02:36.162892 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:36.162897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:36.163064 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:36.163180 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:36.163608 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.164395 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.165369 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:36.165705 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:36.165800 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:36.166141 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:36.166477 (+ 336us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:36.167064 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:36.167197 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 07:02:36.167259 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260619 07:02:36.174348 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 1500
I20260619 07:02:36.181195 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 1575
I20260619 07:02:36.187345 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 1650
2026-06-19T07:02:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3034 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 07:02:36 dist-test-slave-10d0 krb5kdc[1870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 07:02:36 dist-test-slave-10d0 krb5kdc[1870](info): set up 2 sockets
Jun 19 07:02:36 dist-test-slave-10d0 krb5kdc[1870](info): commencing operation
krb5kdc: starting...
W20260619 07:02:37.245709 11689 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/test-user.keytab.
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T07:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T07:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.106.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
Entry for principal kudu/127.11.106.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab.
I20260619 07:02:37.377050 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:32969
--webserver_interface=127.11.106.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:32969,127.11.106.125:40879,127.11.106.124:39703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 07:02:37.491189 1883 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 07:02:37.491539 1883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:37.491590 1883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:37.491663 1883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:37.495698 1883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:37.495787 1883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:37.495827 1883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:37.495862 1883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:37.495898 1883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:37.501194 1883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:32969,127.11.106.125:40879,127.11.106.124:39703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.126.keytab
--principal=kudu/127.11.106.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.126:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.106.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:37.502348 1883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:37.503437 1883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:37.512238 1889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:37.512207 1891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:37.512240 1888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:37.512425 1883 server_base.cc:1034] running on GCE node
I20260619 07:02:37.513190 1883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:37.514354 1883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:37.515522 1883 hybrid_clock.cc:648] HybridClock initialized: now 1781852557515510 us; error 33 us; skew 500 ppm
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:37.518332 1883 init.cc:377] Logged in from keytab as kudu/127.11.106.126@KRBTEST.COM (short username kudu)
I20260619 07:02:37.519294 1883 webserver.cc:460] Webserver started at http://127.11.106.126:40029/ using document root <none> and password file <none>
I20260619 07:02:37.519882 1883 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:37.519974 1883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:37.520215 1883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:37.522392 1883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/data/instance:
uuid: "dc7db02646a4448d94a790532f304b84"
format_stamp: "Formatted at 2026-06-19 07:02:37 on dist-test-slave-10d0"
server_key: "4882996d25eb183b09ded8273181a369"
server_key_iv: "eb9bddd19d0fd7dbe3eefffb7b56949e"
server_key_version: "encryptionkey@0"
I20260619 07:02:37.523027 1883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/wal/instance:
uuid: "dc7db02646a4448d94a790532f304b84"
format_stamp: "Formatted at 2026-06-19 07:02:37 on dist-test-slave-10d0"
server_key: "4882996d25eb183b09ded8273181a369"
server_key_iv: "eb9bddd19d0fd7dbe3eefffb7b56949e"
server_key_version: "encryptionkey@0"
I20260619 07:02:37.526794 1883 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 07:02:37.529294 1898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:37.530483 1883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 07:02:37.530627 1883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/wal
uuid: "dc7db02646a4448d94a790532f304b84"
format_stamp: "Formatted at 2026-06-19 07:02:37 on dist-test-slave-10d0"
server_key: "4882996d25eb183b09ded8273181a369"
server_key_iv: "eb9bddd19d0fd7dbe3eefffb7b56949e"
server_key_version: "encryptionkey@0"
I20260619 07:02:37.530741 1883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:37.549160 1883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:37.552043 1883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:37.552259 1883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:37.559577 1883 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.126:32969
I20260619 07:02:37.559892 1950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.126:32969 every 8 connection(s)
I20260619 07:02:37.560571 1883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/data/info.pb
I20260619 07:02:37.563324 1951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:37.563678 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 1883
I20260619 07:02:37.563829 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-0/wal/instance
I20260619 07:02:37.564133 11689 external_mini_cluster.cc:1428] Setting key 62a8b3470fc1321123f4f20d1bab8943
I20260619 07:02:37.567209 1951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.106.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.125@KRBTEST.COM" created.
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:37.583454 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.569587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:32969 (local address 127.0.0.1:42052)
0619 07:02:37.570072 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.570126 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:37.570970 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:37.571826 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:37.571831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:37.572234 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:37.572492 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.572505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.573763 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.573766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.574609 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.574626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.575468 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.575476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.575622 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:37.576268 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:37.576295 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:37.578324 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:37.580316 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:37.580329 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:37.580341 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:37.580727 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:37.581054 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:37.581059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:37.581062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:37.581240 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:37.581858 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:37.581866 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:37.582048 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 07:02:37.582263 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 07:02:37.582711 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":104,"thread_start_us":144,"threads_started":1}
I20260619 07:02:37.583453 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.569588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.11.106.126:32969)
0619 07:02:37.570167 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:37.570179 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:37.570226 (+ 47us) server_negotiation.cc:407] Connection header received
0619 07:02:37.571309 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:37.571318 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:37.571602 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:37.571812 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:37.572719 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.573643 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.574740 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.575289 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.575411 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:37.578482 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:37.578488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:37.578499 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:37.578521 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:37.580192 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:37.580840 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:37.580845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:37.580847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:37.580953 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:37.581346 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:37.581348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:37.581349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:37.581728 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:37.581832 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:37.582335 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 07:02:37.582609 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":448,"spinlock_wait_cycles":16128,"thread_start_us":159,"threads_started":1}
I20260619 07:02:37.586114 1951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:37.586884 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.586614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:40879 (local address 127.0.0.1:40844)
0619 07:02:37.586740 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.586820 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 07:02:37.587090 1899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
I20260619 07:02:37.587940 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.587730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:40879 (local address 127.0.0.1:40858)
0619 07:02:37.587840 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.587889 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 07:02:37.588102 1951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.125:40879: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab.
Entry for principal kudu/127.11.106.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab.
I20260619 07:02:37.591712 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.125:40879
--webserver_interface=127.11.106.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab
--principal=kudu/127.11.106.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:32969,127.11.106.125:40879,127.11.106.124:39703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:02:37.637357 1951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } attempt: 1
I20260619 07:02:37.638295 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.637972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:40879 (local address 127.0.0.1:40874)
0619 07:02:37.638147 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.638214 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 07:02:37.639050 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.638846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:40879 (local address 127.0.0.1:40880)
0619 07:02:37.638946 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.638991 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 07:02:37.639211 1951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.125:40879: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
W20260619 07:02:37.702687 1958 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 07:02:37.702965 1958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:37.702989 1958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:37.703027 1958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 07:02:37.706957 1958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:37.707067 1958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:37.707108 1958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:37.707145 1958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:37.707175 1958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:37.707381 1951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } attempt: 2
I20260619 07:02:37.708369 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.708009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:40879 (local address 127.0.0.1:40888)
0619 07:02:37.708202 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.708277 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 07:02:37.709200 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.709005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:40879 (local address 127.0.0.1:40900)
0619 07:02:37.709101 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.709146 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 07:02:37.709335 1951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.125:40879: Network error: Client connection negotiation failed: client connection to 127.11.106.125:40879: connect: Connection refused (error 111)
I20260619 07:02:37.713191 1958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:32969,127.11.106.125:40879,127.11.106.124:39703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.125.keytab
--principal=kudu/127.11.106.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.125:40879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.106.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:37.714329 1958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:37.715391 1958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 07:02:37.724087 1958 server_base.cc:1034] running on GCE node
W20260619 07:02:37.724022 1963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:37.724050 1964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:37.724030 1966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:37.724984 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:37.726123 1958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:37.727324 1958 hybrid_clock.cc:648] HybridClock initialized: now 1781852557727306 us; error 31 us; skew 500 ppm
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:37.730057 1958 init.cc:377] Logged in from keytab as kudu/127.11.106.125@KRBTEST.COM (short username kudu)
I20260619 07:02:37.730919 1958 webserver.cc:460] Webserver started at http://127.11.106.125:38227/ using document root <none> and password file <none>
I20260619 07:02:37.731657 1958 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:37.731748 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:37.732017 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:37.734303 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/data/instance:
uuid: "30323a7ccc95457992c2752b2b42643b"
format_stamp: "Formatted at 2026-06-19 07:02:37 on dist-test-slave-10d0"
server_key: "e2207e3ec8fd32bea9631e5fe4bbabf2"
server_key_iv: "a4b749b94fca581db1637cf1cebcfb4a"
server_key_version: "encryptionkey@0"
I20260619 07:02:37.734879 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/wal/instance:
uuid: "30323a7ccc95457992c2752b2b42643b"
format_stamp: "Formatted at 2026-06-19 07:02:37 on dist-test-slave-10d0"
server_key: "e2207e3ec8fd32bea9631e5fe4bbabf2"
server_key_iv: "a4b749b94fca581db1637cf1cebcfb4a"
server_key_version: "encryptionkey@0"
I20260619 07:02:37.738554 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260619 07:02:37.740998 1973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:37.742208 1958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 07:02:37.742354 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/wal
uuid: "30323a7ccc95457992c2752b2b42643b"
format_stamp: "Formatted at 2026-06-19 07:02:37 on dist-test-slave-10d0"
server_key: "e2207e3ec8fd32bea9631e5fe4bbabf2"
server_key_iv: "a4b749b94fca581db1637cf1cebcfb4a"
server_key_version: "encryptionkey@0"
I20260619 07:02:37.742465 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:37.757867 1958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:37.760782 1958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:37.760993 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:37.767763 1958 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.125:40879
I20260619 07:02:37.768062 2026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.125:40879 every 8 connection(s)
I20260619 07:02:37.768787 1958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/data/info.pb
I20260619 07:02:37.771517 2027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:37.775328 2027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:37.778266 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 1958
I20260619 07:02:37.778419 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-1/wal/instance
I20260619 07:02:37.778704 11689 external_mini_cluster.cc:1428] Setting key c80a5414e2d7189483493475ce9181d8
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.106.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.106.124@KRBTEST.COM" created.
I20260619 07:02:37.788767 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.777148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.11.106.126:32969)
0619 07:02:37.777296 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:37.777300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:37.778123 (+ 823us) server_negotiation.cc:407] Connection header received
0619 07:02:37.779170 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:37.779174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:37.779237 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:37.779344 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:37.780483 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.781079 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.782186 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.782426 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.782539 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:37.785150 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:37.785156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:37.785164 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:37.785187 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:37.786641 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:37.787173 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:37.787178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:37.787180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:37.787261 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:37.787613 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:37.787616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:37.787618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:37.787797 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:37.787909 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:37.788430 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 07:02:37.788588 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 07:02:37.789513 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.777503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:32969 (local address 127.0.0.1:42060)
0619 07:02:37.777944 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.777987 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:37.778967 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:37.779514 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:37.779521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:37.779884 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:37.780336 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.780348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.781213 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.781216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.782059 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.782069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.782586 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.782590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.782753 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:37.783351 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:37.783367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:37.784981 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:37.786776 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:37.786782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:37.786796 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:37.787070 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:37.787355 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:37.787359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:37.787361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:37.787499 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:37.787908 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:37.787917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:37.788170 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 07:02:37.788358 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 07:02:37.788714 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
I20260619 07:02:37.791625 2027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
I20260619 07:02:37.800490 1951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } attempt: 3
I20260619 07:02:37.802945 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.792161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:40879 (local address 127.0.0.1:40908)
0619 07:02:37.792604 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.792619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:37.792702 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:37.793070 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:37.793073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:37.793245 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:37.793324 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.793329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.794284 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.794290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.794818 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.794826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.795408 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.795411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.795513 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:37.796002 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:37.796012 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:37.798968 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:37.801206 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:37.801210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:37.801211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:37.801460 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:37.801817 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:37.801823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:37.801826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:37.801912 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:37.802437 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:37.802440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:37.802515 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 07:02:37.802639 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 07:02:37.802757 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":1}
I20260619 07:02:37.803046 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.792265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.11.106.125:40879)
0619 07:02:37.792602 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:37.792616 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:37.792671 (+ 55us) server_negotiation.cc:407] Connection header received
0619 07:02:37.792798 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:37.792802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:37.792953 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:37.793104 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:37.793428 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.794163 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.794929 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.795293 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.795410 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:37.799109 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:37.799118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:37.799130 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:37.799152 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:37.801096 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:37.801607 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:37.801613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:37.801614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:37.801691 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:37.802004 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:37.802007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:37.802009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:37.802321 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:37.802487 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:37.802722 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 07:02:37.802844 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":208,"thread_start_us":133,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.106.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab.
Entry for principal kudu/127.11.106.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab.
I20260619 07:02:37.805002 11689 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1_IlA7/build/debug/bin/kudu
/tmp/dist-test-task1_IlA7/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.124:39703
--webserver_interface=127.11.106.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab
--principal=kudu/127.11.106.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.106.84:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.106.126:32969,127.11.106.125:40879,127.11.106.124:39703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 07:02:37.805106 2027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:37.805846 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.805539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35522)
0619 07:02:37.805704 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.805788 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 07:02:37.806420 1976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
I20260619 07:02:37.807723 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.807427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35532)
0619 07:02:37.807592 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.807653 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 07:02:37.807971 2027 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:39703: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
I20260619 07:02:37.810854 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.801060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:40879 (local address 127.0.0.1:40924)
0619 07:02:37.801228 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.801247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:37.801337 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:37.801786 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:37.801789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:37.801983 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:37.802712 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.802730 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.803718 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.803723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.804473 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.804484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.805184 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.805188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.805292 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:37.805761 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:37.805781 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:37.807353 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:37.809404 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:37.809406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:37.809408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:37.809644 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:37.809964 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:37.809967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:37.809968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:37.810040 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:37.810491 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:37.810493 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:37.810570 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 07:02:37.810648 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 07:02:37.810744 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 07:02:37.811121 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.801121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.11.106.125:40879)
0619 07:02:37.801539 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:37.801554 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:37.801588 (+ 34us) server_negotiation.cc:407] Connection header received
0619 07:02:37.801624 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:37.801626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:37.801660 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:37.801728 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:37.802835 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.803607 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.804630 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.805005 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.805191 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:37.807503 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:37.807506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:37.807509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:37.807539 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:37.809299 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:37.809759 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:37.809766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:37.809771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:37.809869 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:37.810138 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:37.810145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:37.810148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:37.810394 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:37.810512 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:37.810739 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 07:02:37.810869 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":117,"threads_started":1}
I20260619 07:02:37.811760 1951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:37.812435 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.812206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35534)
0619 07:02:37.812319 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.812369 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 07:02:37.813090 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.812860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35548)
0619 07:02:37.812994 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.813038 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 07:02:37.813230 1951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:39703: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
I20260619 07:02:37.844413 1951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } attempt: 1
I20260619 07:02:37.845391 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.845032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35552)
0619 07:02:37.845221 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.845296 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 07:02:37.846171 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.845937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35554)
0619 07:02:37.846065 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.846119 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 07:02:37.846313 1951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:39703: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
I20260619 07:02:37.857228 2027 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } attempt: 1
I20260619 07:02:37.858029 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.857765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35558)
0619 07:02:37.857903 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.857958 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 07:02:37.858747 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.858520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35564)
0619 07:02:37.858644 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.858693 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 07:02:37.858870 2027 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:39703: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
W20260619 07:02:37.918490 2034 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 07:02:37.918761 2034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 07:02:37.918816 2034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 07:02:37.918907 2034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 07:02:37.921489 1951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } attempt: 2
I20260619 07:02:37.922309 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.922004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35574)
0619 07:02:37.922171 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.922232 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 07:02:37.923025 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.922830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35580)
0619 07:02:37.922927 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.922973 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 07:02:37.923177 1951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:39703: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
W20260619 07:02:37.923630 2034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 07:02:37.923679 2034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 07:02:37.923720 2034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 07:02:37.923758 2034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 07:02:37.923792 2034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 07:02:37.927037 2027 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } attempt: 2
I20260619 07:02:37.927703 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.927455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35594)
0619 07:02:37.927582 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.927641 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 07:02:37.928388 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.928183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35610)
0619 07:02:37.928289 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.928334 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 07:02:37.928603 2027 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.106.124:39703: Network error: Client connection negotiation failed: client connection to 127.11.106.124:39703: connect: Connection refused (error 111)
I20260619 07:02:37.929430 2034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.106.84:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.106.126:32969,127.11.106.125:40879,127.11.106.124:39703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/krb5kdc/kudu_127.11.106.124.keytab
--principal=kudu/127.11.106.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.106.124:39703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.106.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 06:14:33 UTC on ccycloud.quasar-ryanpe.root.comops.site
build id 4455156
I20260619 07:02:37.930434 2034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 07:02:37.931473 2034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 07:02:37.940011 2043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 07:02:37.940033 2041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:37.940258 2034 server_base.cc:1034] running on GCE node
W20260619 07:02:37.940033 2040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 07:02:37.940909 2034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 07:02:37.941988 2034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 07:02:37.943198 2034 hybrid_clock.cc:648] HybridClock initialized: now 1781852557943179 us; error 27 us; skew 500 ppm
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 07:02:37.945982 2034 init.cc:377] Logged in from keytab as kudu/127.11.106.124@KRBTEST.COM (short username kudu)
I20260619 07:02:37.946995 2034 webserver.cc:460] Webserver started at http://127.11.106.124:45775/ using document root <none> and password file <none>
I20260619 07:02:37.947569 2034 fs_manager.cc:359] Metadata directory not provided
I20260619 07:02:37.947664 2034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 07:02:37.947893 2034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 07:02:37.949947 2034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/data/instance:
uuid: "e3b43a129a064024b93fe1d8dc417b89"
format_stamp: "Formatted at 2026-06-19 07:02:37 on dist-test-slave-10d0"
server_key: "a3163aaa56a490549c5cf0c8cb3c34d5"
server_key_iv: "bf4e85ee9ea6ac7dfedb6af0c22f286a"
server_key_version: "encryptionkey@0"
I20260619 07:02:37.950520 2034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/wal/instance:
uuid: "e3b43a129a064024b93fe1d8dc417b89"
format_stamp: "Formatted at 2026-06-19 07:02:37 on dist-test-slave-10d0"
server_key: "a3163aaa56a490549c5cf0c8cb3c34d5"
server_key_iv: "bf4e85ee9ea6ac7dfedb6af0c22f286a"
server_key_version: "encryptionkey@0"
I20260619 07:02:37.954190 2034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 07:02:37.956491 2050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 07:02:37.957732 2034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 07:02:37.957877 2034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/data,/tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/wal
uuid: "e3b43a129a064024b93fe1d8dc417b89"
format_stamp: "Formatted at 2026-06-19 07:02:37 on dist-test-slave-10d0"
server_key: "a3163aaa56a490549c5cf0c8cb3c34d5"
server_key_iv: "bf4e85ee9ea6ac7dfedb6af0c22f286a"
server_key_version: "encryptionkey@0"
I20260619 07:02:37.957985 2034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 07:02:37.974174 2034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 07:02:37.977252 2034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 07:02:37.977454 2034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 07:02:37.984498 2034 rpc_server.cc:307] RPC server started. Bound to: 127.11.106.124:39703
I20260619 07:02:37.984792 2102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.106.124:39703 every 8 connection(s)
I20260619 07:02:37.985633 2034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/data/info.pb
I20260619 07:02:37.988394 2103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 07:02:37.992267 11689 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1_IlA7/build/debug/bin/kudu as pid 2034
I20260619 07:02:37.992075 2103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:37.992408 11689 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1_IlA7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781852273483449-11689-0/minicluster-data/master-2/wal/instance
I20260619 07:02:37.992779 11689 external_mini_cluster.cc:1428] Setting key 893c10807c8eba7eb676dae2e1161eff
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
Jun 19 07:02:37 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
I20260619 07:02:38.005729 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.994866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.11.106.126:32969)
0619 07:02:37.995023 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:37.995027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:37.995240 (+ 213us) server_negotiation.cc:407] Connection header received
0619 07:02:37.995692 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:37.995695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:37.995741 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:37.995855 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:37.996801 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.997292 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.998413 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.998748 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.998828 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.001760 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.001766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.001770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.001804 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.003334 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.003903 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.003905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.003906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.003979 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.004352 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.004357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.004359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.004662 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.004755 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.005404 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.005535 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 07:02:38.006879 2107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.994710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.126:32969 (local address 127.0.0.1:42062)
0619 07:02:37.995092 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:37.995142 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:37.995451 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:37.995917 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:37.995923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:37.996335 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:37.996645 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.996667 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.997400 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.997404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.998292 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:37.998301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.998883 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.998889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:37.999131 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:37.999807 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:37.999823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:38.001525 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:38.003452 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.003461 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.003472 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.003800 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.004064 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.004067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.004078 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.004259 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.004763 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:38.004769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:38.005112 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 07:02:38.005331 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 07:02:38.005681 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":45,"thread_start_us":132,"threads_started":1}
I20260619 07:02:38.006880 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:37.994088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.11.106.124:39703)
0619 07:02:37.994491 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:37.994508 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:37.994542 (+ 34us) server_negotiation.cc:407] Connection header received
0619 07:02:37.995433 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:37.995454 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:37.995696 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:37.995873 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:37.996677 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.997687 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.998685 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:37.999212 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:37.999327 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.002160 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.002168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.002183 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.002212 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.004156 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.004659 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.004664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.004665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.004760 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.005026 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.005030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.005031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.005285 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.005492 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.005742 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.005891 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260619 07:02:38.008445 2103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 07:02:38 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
Jun 19 07:02:38 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.126@KRBTEST.COM
I20260619 07:02:38.017578 2107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.008982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.125:40879 (local address 127.0.0.1:40936)
0619 07:02:38.009131 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:38.009147 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:38.009221 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:38.009463 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:38.009468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:38.009669 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:38.009799 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:38.009807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.010814 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.010817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:38.011454 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:38.011462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.012095 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.012098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:38.012200 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.012777 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:38.012788 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:38.014074 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:38.016055 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.016058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.016061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.016276 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.016587 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.016593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.016597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.016687 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.017060 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:38.017063 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:38.017168 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 07:02:38.017274 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 07:02:38.017404 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 07:02:38.017599 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.009120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.11.106.125:40879)
0619 07:02:38.009242 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.009249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.009265 (+ 16us) server_negotiation.cc:407] Connection header received
0619 07:02:38.009309 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.009312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.009363 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.009442 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.009937 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.010711 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.011566 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.011959 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.012045 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.014208 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.014211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.014213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.014236 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.015935 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.016378 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.016382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.016384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.016464 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.016781 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.016785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.016787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.016963 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.017070 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.017326 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.017449 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 07:02:38.018414 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.009328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.11.106.126:32969)
0619 07:02:38.009444 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.009448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.009460 (+ 12us) server_negotiation.cc:407] Connection header received
0619 07:02:38.009617 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.009621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.009668 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.009751 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.010476 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.011062 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.011947 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.012253 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.012375 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.014870 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.014874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.014877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.014907 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.016398 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.016849 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.016854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.016855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.016936 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.017220 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.017224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.017225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.017392 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.018033 (+ 641us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.018155 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.018251 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 07:02:38.018366 2103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } has no permanent_uuid. Determining permanent_uuid...
I20260619 07:02:38.019749 2027 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } attempt: 3
I20260619 07:02:38.022351 1951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } attempt: 3
Jun 19 07:02:38 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.124@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
Jun 19 07:02:38 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.125@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
Jun 19 07:02:38 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.106.125@KRBTEST.COM
Jun 19 07:02:38 dist-test-slave-10d0 krb5kdc[1870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781852557, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.106.126@KRBTEST.COM for kudu/127.11.106.124@KRBTEST.COM
I20260619 07:02:38.029896 2107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.018900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35638)
0619 07:02:38.018989 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:38.019004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:38.019077 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:38.019312 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:38.019314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:38.019477 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:38.019587 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:38.019596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.020503 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.020507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:38.021240 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:38.021249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.021703 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.021706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:38.021828 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.022282 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:38.022296 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:38.023690 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:38.027989 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.027991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.027993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.028396 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.028728 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.028731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.028732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.028816 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.029376 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:38.029379 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:38.029468 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 07:02:38.029574 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 07:02:38.029704 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":35}
I20260619 07:02:38.030442 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.018973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.11.106.124:39703)
0619 07:02:38.019097 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.019104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.019118 (+ 14us) server_negotiation.cc:407] Connection header received
0619 07:02:38.019167 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.019169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.019211 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.019280 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.019707 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.020406 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.021352 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.021601 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.021706 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.023855 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.023860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.023863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.023885 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.027869 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.028508 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.028511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.028513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.028625 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.028914 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.028917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.028918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.029257 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.029345 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.030110 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.030273 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":52}
I20260619 07:02:38.030876 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.020117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35642)
0619 07:02:38.020270 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:38.020282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:38.020346 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:38.020738 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:38.020741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:38.020901 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:38.021013 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:38.021019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.022390 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.022394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:38.023134 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:38.023145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.023772 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.023777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:38.023878 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.024328 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:38.024347 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:38.025995 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:38.028733 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.028737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.028739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.029154 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.029495 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.029499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.029502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.029615 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.030380 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:38.030383 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:38.030490 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 07:02:38.030589 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 07:02:38.030726 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1}
I20260619 07:02:38.031139 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.019704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.11.106.125:40879)
0619 07:02:38.019847 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.019851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.019864 (+ 13us) server_negotiation.cc:407] Connection header received
0619 07:02:38.020040 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.020044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.020086 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.020158 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.020974 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.021728 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.023309 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.023654 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.023763 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.027130 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.027136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.027142 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.027170 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.029232 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.029731 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.029734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.029735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.029810 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.030111 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.030115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.030118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.030320 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.030539 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.030865 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.030977 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":48}
I20260619 07:02:38.032007 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.020176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.11.106.124:39703)
0619 07:02:38.020503 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.020513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.020524 (+ 11us) server_negotiation.cc:407] Connection header received
0619 07:02:38.020579 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.020582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.020617 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.020719 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.021135 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.022270 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.023283 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.023650 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.024082 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.026349 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.026354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.026357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.026411 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.028607 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.029272 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.029279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.029283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.029376 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.029747 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.029751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.029755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.030253 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.031644 (+ 1391us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.031721 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.031831 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
I20260619 07:02:38.033972 2103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89: Bootstrap starting.
I20260619 07:02:38.034633 11689 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 07:02:38.035027 2027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b: Bootstrap starting.
I20260619 07:02:38.036656 2103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:38.035936 1925 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dc7db02646a4448d94a790532f304b84"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42072
I20260619 07:02:38.037400 2027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:38.037560 2103 log.cc:824] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:38.038251 2027 log.cc:824] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:38.039744 2103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89: No bootstrap required, opened a new log
I20260619 07:02:38.040584 1954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.022721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.106.124:39703 (local address 127.0.0.1:35644)
0619 07:02:38.022846 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 07:02:38.022861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 07:02:38.022938 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 07:02:38.023699 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 07:02:38.023702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 07:02:38.023880 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 07:02:38.023991 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:38.024001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.024975 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.024978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:38.025488 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 07:02:38.025496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.026021 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.026025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 07:02:38.026115 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.026544 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 07:02:38.026561 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 07:02:38.028736 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 07:02:38.038470 (+ 9734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.038476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.038478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.038776 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.039141 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 07:02:38.039145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 07:02:38.039147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 07:02:38.039248 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 07:02:38.039991 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 07:02:38.039994 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 07:02:38.040114 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 07:02:38.040234 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 07:02:38.040392 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 07:02:38.040627 2027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b: No bootstrap required, opened a new log
I20260619 07:02:38.041170 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.022875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.11.106.124:39703)
0619 07:02:38.023469 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.023478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.023490 (+ 12us) server_negotiation.cc:407] Connection header received
0619 07:02:38.023519 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.023522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.023560 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.024113 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.024215 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.024859 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.025628 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.025898 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.026006 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.028873 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.028877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.028882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.028907 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.038322 (+ 9415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.038920 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.038925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.038929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.039021 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.039371 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.039374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.039376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.039743 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.040010 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.040381 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.040622 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":94,"threads_started":1}
I20260619 07:02:38.044265 2103 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } }
I20260619 07:02:38.044476 1951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84: Bootstrap starting.
I20260619 07:02:38.044523 2103 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b43a129a064024b93fe1d8dc417b89, State: Initialized, Role: FOLLOWER
I20260619 07:02:38.044727 2027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } }
I20260619 07:02:38.044939 2027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30323a7ccc95457992c2752b2b42643b, State: Initialized, Role: FOLLOWER
I20260619 07:02:38.044885 2103 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } }
I20260619 07:02:38.045394 2027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } }
I20260619 07:02:38.045909 2112 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.046233 2112 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:38.046577 2103 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:38.046752 2027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:38.047452 1951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84: Neither blocks nor log segments found. Creating new log.
I20260619 07:02:38.048380 1951 log.cc:824] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84: Log is configured to *not* fsync() on all Append() calls
I20260619 07:02:38.051319 1951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84: No bootstrap required, opened a new log
W20260619 07:02:38.052273 2131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:38.052361 2131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:38.050519 1958 master_runner.cc:186] Error getting master registration for 127.11.106.126:32969: OK, instance_id { permanent_uuid: "dc7db02646a4448d94a790532f304b84" instance_seqno: 1781852557557545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 07:02:38.054468 1958 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 07:02:38.054594 1951 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } }
I20260619 07:02:38.054804 1951 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7db02646a4448d94a790532f304b84, State: Initialized, Role: FOLLOWER
I20260619 07:02:38.055141 1951 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } }
I20260619 07:02:38.055187 2113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.055349 2113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:38.056440 1951 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 07:02:38.056489 1925 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 07:02:38.056610 1925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:38.057906 1925 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } }
I20260619 07:02:38.059427 1925 leader_election.cc:288] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [CANDIDATE]: Term 1 election: Requested vote from peers 30323a7ccc95457992c2752b2b42643b (127.11.106.125:40879), e3b43a129a064024b93fe1d8dc417b89 (127.11.106.124:39703)
I20260619 07:02:38.059456 2001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc7db02646a4448d94a790532f304b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "30323a7ccc95457992c2752b2b42643b"
W20260619 07:02:38.059610 2136 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:38.059664 2136 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:38.059695 2001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:38.056574 2034 master_runner.cc:186] Error getting master registration for 127.11.106.126:32969: OK, instance_id { permanent_uuid: "dc7db02646a4448d94a790532f304b84" instance_seqno: 1781852557557545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 07:02:38.059913 2034 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 07:02:38.060519 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc7db02646a4448d94a790532f304b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e3b43a129a064024b93fe1d8dc417b89"
I20260619 07:02:38.060797 2077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 07:02:38.060901 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.061048 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:38.062310 2077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc7db02646a4448d94a790532f304b84 in term 1.
I20260619 07:02:38.062834 1901 leader_election.cc:302] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc7db02646a4448d94a790532f304b84, e3b43a129a064024b93fe1d8dc417b89; no voters:
I20260619 07:02:38.063105 2138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 07:02:38.063310 2138 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [term 1 LEADER]: Becoming Leader. State: Replica: dc7db02646a4448d94a790532f304b84, State: Running, Role: LEADER
I20260619 07:02:38.063321 2001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc7db02646a4448d94a790532f304b84 in term 1.
I20260619 07:02:38.063616 2138 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } }
I20260619 07:02:38.065415 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc7db02646a4448d94a790532f304b84. Latest consensus state: current_term: 1 leader_uuid: "dc7db02646a4448d94a790532f304b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.065521 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [sys.catalog]: This master's current role is: LEADER
W20260619 07:02:38.065862 2154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 07:02:38.065936 2154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 07:02:38.070484 2158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 07:02:38.072685 2158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 07:02:38.078696 2077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [term 1 FOLLOWER]: Refusing update from remote peer dc7db02646a4448d94a790532f304b84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:38.079442 2155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:38.083637 2112 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc7db02646a4448d94a790532f304b84. Latest consensus state: current_term: 1 leader_uuid: "dc7db02646a4448d94a790532f304b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.083810 2112 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:38.084007 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.069461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.11.106.124:39703)
0619 07:02:38.069596 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.069601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.071017 (+ 1416us) server_negotiation.cc:407] Connection header received
0619 07:02:38.071217 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.071221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.071283 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.071338 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.072261 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.073027 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.074225 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.074616 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.074696 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.076635 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.076639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.076643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.076667 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.078415 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.081297 (+ 2882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.081301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.081303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.081400 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.081719 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.081723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.081725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.081982 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.082109 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.083414 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.083547 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 07:02:38.086071 2001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [term 1 FOLLOWER]: Refusing update from remote peer dc7db02646a4448d94a790532f304b84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 07:02:38.086684 2155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 07:02:38.090971 2113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc7db02646a4448d94a790532f304b84. Latest consensus state: current_term: 1 leader_uuid: "dc7db02646a4448d94a790532f304b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.091109 2113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:38.092708 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.071295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.11.106.125:40879)
0619 07:02:38.071518 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.071522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.084171 (+ 12649us) server_negotiation.cc:407] Connection header received
0619 07:02:38.084344 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.084348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.084406 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.084458 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.085466 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.086227 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.087288 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.087621 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.087690 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.089257 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.089261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.089264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.089291 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.090947 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.091427 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.091431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.091433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.091513 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.091786 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.091789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.091791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.091990 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.092088 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.092408 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.092521 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260619 07:02:38.093281 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:38.069857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.11.106.126:32969)
0619 07:02:38.069971 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.069976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.076942 (+ 6966us) server_negotiation.cc:407] Connection header received
0619 07:02:38.076984 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.076987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.077034 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.077106 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.079081 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.079791 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.082445 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:38.082791 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:38.082871 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:38.086924 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 07:02:38.086928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 07:02:38.086931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 07:02:38.086960 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 07:02:38.089071 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.090449 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.090454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.090456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.090557 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 07:02:38.091224 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 07:02:38.091228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 07:02:38.091230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 07:02:38.091498 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 07:02:38.091643 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:38.092997 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0619 07:02:38.093129 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 07:02:38.095937 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc7db02646a4448d94a790532f304b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.096063 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:38.096321 2112 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc7db02646a4448d94a790532f304b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.096390 2162 mvcc.cc:205] Tried to move back new op lower bound from 7298468077881774080 to 7298468077831892992. Current Snapshot: MvccSnapshot[applied={T|T < 7298468077881774080}]
I20260619 07:02:38.096452 2112 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:38.096984 2163 mvcc.cc:205] Tried to move back new op lower bound from 7298468077881774080 to 7298468077831892992. Current Snapshot: MvccSnapshot[applied={T|T < 7298468077881774080}]
I20260619 07:02:38.098549 2155 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc7db02646a4448d94a790532f304b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.098663 2155 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84 [sys.catalog]: This master's current role is: LEADER
I20260619 07:02:38.100353 2113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc7db02646a4448d94a790532f304b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7db02646a4448d94a790532f304b84" member_type: VOTER last_known_addr { host: "127.11.106.126" port: 32969 } } peers { permanent_uuid: "30323a7ccc95457992c2752b2b42643b" member_type: VOTER last_known_addr { host: "127.11.106.125" port: 40879 } } peers { permanent_uuid: "e3b43a129a064024b93fe1d8dc417b89" member_type: VOTER last_known_addr { host: "127.11.106.124" port: 39703 } } }
I20260619 07:02:38.100474 2113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 07:02:38.103297 2158 catalog_manager.cc:1347] Generated new cluster ID: 02880b34405e4d7ab90e589e7f2dfefa
I20260619 07:02:38.103358 2158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 07:02:38.104909 2165 mvcc.cc:205] Tried to move back new op lower bound from 7298468077881774080 to 7298468077831892992. Current Snapshot: MvccSnapshot[applied={T|T < 7298468077881774080 or (T in {7298468077881774080})}]
I20260619 07:02:38.131464 2158 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 07:02:38.133047 2158 catalog_manager.cc:1504] Loading token signing keys...
I20260619 07:02:38.152290 2158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc7db02646a4448d94a790532f304b84: Generated new TSK 0
I20260619 07:02:38.153349 2158 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 07:02:38.189735 1955 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.186870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.11.106.126:32969)
0619 07:02:38.187017 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.187022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.188457 (+ 1435us) server_negotiation.cc:407] Connection header received
0619 07:02:38.188710 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.188714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.188776 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.188836 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.189586 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42084: BlockingRecv error: recv got EOF from 127.0.0.1:42084 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 07:02:38.191622 2109 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.187791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.11.106.124:39703)
0619 07:02:38.187957 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.187963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.190338 (+ 2375us) server_negotiation.cc:407] Connection header received
0619 07:02:38.190730 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.190734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.190791 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.190849 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.191480 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35656: BlockingRecv error: recv got EOF from 127.0.0.1:35656 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 07:02:38.193753 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.192281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.11.106.125:40879)
0619 07:02:38.192423 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.192428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.192796 (+ 368us) server_negotiation.cc:407] Connection header received
0619 07:02:38.192980 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.192983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.193033 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.193084 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.193624 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40948: BlockingRecv error: recv got EOF from 127.0.0.1:40948 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 07:02:38.198791 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.197115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.11.106.125:40879)
0619 07:02:38.197225 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.197230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.197691 (+ 461us) server_negotiation.cc:407] Connection header received
0619 07:02:38.197936 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.197940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.198029 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.198111 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.198675 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40954: BlockingRecv error: recv got EOF from 127.0.0.1:40954 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 07:02:38.204460 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.202810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.11.106.125:40879)
0619 07:02:38.202925 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.202930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.203462 (+ 532us) server_negotiation.cc:407] Connection header received
0619 07:02:38.203668 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.203672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.203727 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.203783 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.204335 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40956: BlockingRecv error: recv got EOF from 127.0.0.1:40956 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 07:02:38.230501 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.221144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.11.106.125:40879)
0619 07:02:38.221303 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.221310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.229087 (+ 7777us) server_negotiation.cc:407] Connection header received
0619 07:02:38.229373 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.229380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.229461 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.229545 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.230343 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40970: BlockingRecv error: recv got EOF from 127.0.0.1:40970 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 07:02:38.247880 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.241034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40978 (local address 127.11.106.125:40879)
0619 07:02:38.241232 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.241239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.246624 (+ 5385us) server_negotiation.cc:407] Connection header received
0619 07:02:38.246849 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.246854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.246944 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.247059 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.247747 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40978: BlockingRecv error: recv got EOF from 127.0.0.1:40978 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 07:02:38.270152 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.265171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.11.106.125:40879)
0619 07:02:38.265336 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.265342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.269139 (+ 3797us) server_negotiation.cc:407] Connection header received
0619 07:02:38.269213 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.269216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.269282 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.269374 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.270012 (+ 638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40982: BlockingRecv error: recv got EOF from 127.0.0.1:40982 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 07:02:38.290762 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.288908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.11.106.125:40879)
0619 07:02:38.289041 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.289046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.289723 (+ 677us) server_negotiation.cc:407] Connection header received
0619 07:02:38.289950 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.289954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.290011 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.290066 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.290634 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40996: BlockingRecv error: recv got EOF from 127.0.0.1:40996 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 07:02:38.326537 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.325392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.11.106.125:40879)
0619 07:02:38.325542 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.325547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.325709 (+ 162us) server_negotiation.cc:407] Connection header received
0619 07:02:38.325818 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.325820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.325867 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.325941 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.326394 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41002: BlockingRecv error: recv got EOF from 127.0.0.1:41002 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 07:02:38.394029 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.392861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.11.106.125:40879)
0619 07:02:38.393024 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.393031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.393184 (+ 153us) server_negotiation.cc:407] Connection header received
0619 07:02:38.393289 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.393292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.393342 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.393417 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.393896 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41018: BlockingRecv error: recv got EOF from 127.0.0.1:41018 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 07:02:38.525466 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.524241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.11.106.125:40879)
0619 07:02:38.524374 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.524378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.524625 (+ 247us) server_negotiation.cc:407] Connection header received
0619 07:02:38.524740 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.524743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.524793 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.524866 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.525310 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41020: BlockingRecv error: recv got EOF from 127.0.0.1:41020 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 07:02:38.784677 2033 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 07:02:38.783665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.11.106.125:40879)
0619 07:02:38.783810 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:38.783813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:38.783829 (+ 16us) server_negotiation.cc:407] Connection header received
0619 07:02:38.783910 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:38.783913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:38.783958 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:38.784030 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 07:02:38.784546 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41032: BlockingRecv error: recv got EOF from 127.0.0.1:41032 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 07:02:39.053715 2131 catalog_manager.cc:1259] Loaded cluster ID: 02880b34405e4d7ab90e589e7f2dfefa
I20260619 07:02:39.053800 2131 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89: loading cluster ID for follower catalog manager: success
I20260619 07:02:39.055403 2131 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89: acquiring CA information for follower catalog manager: success
I20260619 07:02:39.056799 2131 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e3b43a129a064024b93fe1d8dc417b89: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 07:02:39.061270 2136 catalog_manager.cc:1259] Loaded cluster ID: 02880b34405e4d7ab90e589e7f2dfefa
I20260619 07:02:39.061353 2136 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b: loading cluster ID for follower catalog manager: success
I20260619 07:02:39.063114 2136 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b: acquiring CA information for follower catalog manager: success
I20260619 07:02:39.064214 2136 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30323a7ccc95457992c2752b2b42643b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 07:02:39.302879 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:39.298997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.11.106.125:40879)
0619 07:02:39.299255 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:39.299258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:39.299276 (+ 18us) server_negotiation.cc:407] Connection header received
0619 07:02:39.299378 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:39.299380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:39.299507 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:39.299610 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:39.299843 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.300373 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.301283 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.301578 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.301698 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:39.302000 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:39.302492 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:39.302605 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:39.302661 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 07:02:39.302724 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260619 07:02:39.310156 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:39.306258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.11.106.124:39703)
0619 07:02:39.306537 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:39.306541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:39.306557 (+ 16us) server_negotiation.cc:407] Connection header received
0619 07:02:39.306649 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:39.306652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:39.306778 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:39.306901 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:39.307109 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.307715 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.308627 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.308900 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.309005 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:39.309283 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:39.309788 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:39.309904 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:39.309966 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 07:02:39.310019 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":116,"threads_started":1}
I20260619 07:02:39.317221 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:39.313458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.11.106.125:40879)
0619 07:02:39.313609 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:39.313615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:39.313745 (+ 130us) server_negotiation.cc:407] Connection header received
0619 07:02:39.313939 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:39.313945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:39.314009 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:39.314104 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:39.314326 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.315133 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.316087 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.316306 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.316415 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:39.316725 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:39.316804 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:39.316878 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:39.317033 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 07:02:39.317094 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 07:02:39.317873 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:39.313305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.11.106.124:39703)
0619 07:02:39.313437 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:39.313442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:39.313930 (+ 488us) server_negotiation.cc:407] Connection header received
0619 07:02:39.314009 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:39.314011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:39.314070 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:39.314141 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:39.314479 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.315223 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.316593 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.316934 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.317028 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:39.317376 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:39.317447 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:39.317505 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:39.317674 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 07:02:39.317738 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 07:02:39.318127 2248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 07:02:39.313287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.11.106.126:32969)
0619 07:02:39.313555 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 07:02:39.313559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 07:02:39.313794 (+ 235us) server_negotiation.cc:407] Connection header received
0619 07:02:39.314078 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 07:02:39.314082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 07:02:39.314249 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 07:02:39.314367 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 07:02:39.314695 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.315302 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.316565 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 07:02:39.316867 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 07:02:39.316973 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 07:02:39.317320 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 07:02:39.317679 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 07:02:39.317765 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 07:02:39.317908 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 07:02:39.317971 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260619 07:02:39.321498 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 1883
I20260619 07:02:39.328869 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 1958
I20260619 07:02:39.335040 11689 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1_IlA7/build/debug/bin/kudu with pid 2034
2026-06-19T07:02:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3147 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6182 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285840 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST